Page MenuHomePhabricator
Create Task
Maniphest T368289

Incredible amount of logs from Wikimedia\Rdbms\LoadBalancer::runPrimaryTransactionIdleCallbacks
Open, In Progress, HighPublic
Actions

Description

During connectivity issues in production, these logs are generated at incredible volumes.

June 22 2024 22:00 UTC to June 23 2024 05:30 UTC for message:"found writes pending"

CountNormalized message
18,375,432Wikimedia\Rdbms\LoadBalancer::runPrimaryTransactionIdleCallbacks: found writes pending ().
11,027,272Wikimedia\Rdbms\LoadBalancer::runPrimaryTransactionIdleCallbacks: found writes pending (MediaWiki\Deferred\LinksUpdate\LinksUpdate::acquirePageLock).
5,145,993Wikimedia\Rdbms\LoadBalancer::runPrimaryTransactionIdleCallbacks: found writes pending (CategoryMembershipChangeJob::run).
2,841,780Wikimedia\Rdbms\LoadBalancer::runPrimaryTransactionIdleCallbacks: found writes pending (MediaWiki\Linter\Database::setForPage).
2,167,911Wikimedia\Rdbms\LoadBalancer::runPrimaryTransactionIdleCallbacks: found writes pending (MediaWiki\Deferred\LinksUpdate\LinksUpdate::updateLinksTimestamp, MediaWiki\Deferred\LinksUpdate\LinksUpdate::acquirePageLock, MediaWiki\Deferred\LinksUpdate\LinksUpdate
......

Details

SubjectRepoBranchLines +/-
rdbms: Reduce log severity of "found writes pending"mediawiki/corewmf/1.43.0-wmf.11+1 -1
rdbms: Reduce log severity of "found writes pending"mediawiki/coremaster+1 -1
rdbms: Reduce log severity of "found writes pending"mediawiki/corewmf/1.43.0-wmf.10+1 -1
Customize query in gerrit

Related Objects
Search...

Event Timeline

colewhite created this task.Mon, Jun 24, 3:46 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMon, Jun 24, 3:46 PM
colewhite triaged this task as High priority.Mon, Jun 24, 3:46 PM
colewhite added a parent task: T368098: Dumps generation without prefetch cause disruption to the production environment.
gerritbot added a comment.Mon, Jun 24, 4:24 PM

Change #1049228 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[mediawiki/core@master] rdbms: Reduce log severity of "found writes pending"

https://gerrit.wikimedia.org/r/1049228

gerritbot added a project: Patch-For-Review.Mon, Jun 24, 4:24 PM
hashar updated the task description. (Show Details)Mon, Jun 24, 4:27 PM
Ladsgroup claimed this task.Mon, Jun 24, 4:31 PM
Ladsgroup added a project: MediaWiki-libs-Rdbms.
aaron added a project: MW-Interfaces-Team.Mon, Jun 24, 5:39 PM
aaron changed the task status from Open to In Progress.Mon, Jun 24, 5:54 PM
aaron moved this task from Incoming (Needs Triage) to In Progress on the MW-Interfaces-Team board.
gerritbot added a comment.Mon, Jun 24, 8:11 PM

Change #1049261 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[mediawiki/core@wmf/1.43.0-wmf.10] rdbms: Reduce log severity of "found writes pending"

https://gerrit.wikimedia.org/r/1049261

gerritbot added a comment.Mon, Jun 24, 8:34 PM

Change #1049261 merged by jenkins-bot:

[mediawiki/core@wmf/1.43.0-wmf.10] rdbms: Reduce log severity of "found writes pending"

https://gerrit.wikimedia.org/r/1049261

ReleaseTaggerBot added a project: MW-1.43-notes (1.43.0-wmf.10; 2024-06-18).Mon, Jun 24, 9:00 PM
MatthewVernon subscribed.Tue, Jun 25, 10:25 AM
dr0ptp4kt subscribed.Tue, Jun 25, 6:27 PM
gerritbot added a comment.Tue, Jun 25, 8:37 PM

Change #1049228 merged by jenkins-bot:

[mediawiki/core@master] rdbms: Reduce log severity of "found writes pending"

https://gerrit.wikimedia.org/r/1049228

ReleaseTaggerBot edited projects, added MW-1.43-notes (1.43.0-wmf.12; 2024-07-02); removed MW-1.43-notes (1.43.0-wmf.10; 2024-06-18).Tue, Jun 25, 9:00 PM
Maintenance_bot removed a project: Patch-For-Review.Tue, Jun 25, 9:30 PM
gerritbot added a comment.Wed, Jun 26, 11:47 AM

Change #1049909 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[mediawiki/core@wmf/1.43.0-wmf.11] rdbms: Reduce log severity of "found writes pending"

https://gerrit.wikimedia.org/r/1049909

gerritbot added a project: Patch-For-Review.Wed, Jun 26, 11:47 AM
gerritbot added a comment.Wed, Jun 26, 12:15 PM

Change #1049909 merged by jenkins-bot:

[mediawiki/core@wmf/1.43.0-wmf.11] rdbms: Reduce log severity of "found writes pending"

https://gerrit.wikimedia.org/r/1049909

Maintenance_bot removed a project: Patch-For-Review.Wed, Jun 26, 12:31 PM
xcollazo subscribed.Wed, Jun 26, 12:34 PM
ReleaseTaggerBot edited projects, added MW-1.43-notes (1.43.0-wmf.11; 2024-06-25); removed MW-1.43-notes (1.43.0-wmf.12; 2024-07-02).Wed, Jun 26, 1:00 PM
Ladsgroup added a comment.Wed, Jun 26, 3:47 PM

I've been trying to debug and demystify what caused this. Here are some preliminary notes:

colewhite added a comment.EditedWed, Jun 26, 5:59 PM

FTR, after https://gerrit.wikimedia.org/r/c/operations/puppet/+/1048863 we dropped 99% of "found writes pending" logs randomly as they appeared from Kafka. I have deleted no ingested logs.

According to the Kafka topic, we got around 560,778 logs per second (src). This doesn't specifically select just "found writes pending" logs, but the udp_localhost-warning topic doesn't get a lot of traffic.

Ladsgroup added a comment.Fri, Jun 28, 6:44 PM

Given that 99% of the logs were dropped, it mean each job triggered 1000 "writes pending found" log. The only explanation is that it fell into an infinite loop when databases got lagged or slow. That part is not fixed yet and should be investigated further.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits