Page MenuHomePhabricator

Transaction should be in the callback stage (not 'cursory')
Closed, ResolvedPublic

Description

Saw a flood of these errors in fatalmonitor following the rollout of 1.32.0-wmf.2 to group1

Stacktrace:

#0 /srv/mediawiki/php-1.32.0-wmf.2/includes/libs/rdbms/lbfactory/LBFactory.php(267): Wikimedia\Rdbms\LoadBalancer->runMasterTransactionListenerCallbacks()
#1 [internal function]: Closure$Wikimedia\Rdbms\LBFactory::commitMasterChanges#2(Wikimedia\Rdbms\LoadBalancer)
#2 /srv/mediawiki/php-1.32.0-wmf.2/includes/libs/rdbms/lbfactory/LBFactoryMulti.php(425): call_user_func_array(Closure$Wikimedia\Rdbms\LBFactory::commitMasterChanges#2;4828, array)
#3 /srv/mediawiki/php-1.32.0-wmf.2/includes/libs/rdbms/lbfactory/LBFactory.php(269): Wikimedia\Rdbms\LBFactoryMulti->forEachLB(Closure$Wikimedia\Rdbms\LBFactory::commitMasterChanges#2;4828)
#4 /srv/mediawiki/php-1.32.0-wmf.2/includes/libs/rdbms/lbfactory/LBFactory.php(445): Wikimedia\Rdbms\LBFactory->commitMasterChanges(string)
#5 /srv/mediawiki/php-1.32.0-wmf.2/includes/jobqueue/jobs/RefreshLinksJob.php(290): Wikimedia\Rdbms\LBFactory->commitAndWaitForReplication(string, integer)
#6 /srv/mediawiki/php-1.32.0-wmf.2/includes/jobqueue/jobs/RefreshLinksJob.php(122): RefreshLinksJob->runForTitle(Title)
#7 /srv/mediawiki/php-1.32.0-wmf.2/extensions/EventBus/includes/JobExecutor.php(60): RefreshLinksJob->run()
#8 /srv/mediawiki/rpc/RunSingleJob.php(79): JobExecutor->execute(array)
#9 {main}

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 2 2018, 7:28 PM
thcipriani triaged this task as Unbreak Now! priority.May 2 2018, 7:29 PM

Added as train blocker setting as UBN!

Restricted Application added subscribers: Liuxinyu970226, TerraCodes. · View Herald TranscriptMay 2 2018, 7:30 PM

Adding @aaron since it looks like the messages were added in https://gerrit.wikimedia.org/r/#/c/422543/

Change 430507 had a related patch set uploaded (by Aaron Schulz; owner: Aaron Schulz):
[mediawiki/core@wmf/1.32.0-wmf.2] Revert "rdbms: enforce and improve LBFactory/LoadBalancer callback handling"

https://gerrit.wikimedia.org/r/430507

Change 430507 merged by jenkins-bot:
[mediawiki/core@wmf/1.32.0-wmf.2] Revert "rdbms: enforce and improve LBFactory/LoadBalancer callback handling"

https://gerrit.wikimedia.org/r/430507

Change 430532 had a related patch set uploaded (by Aaron Schulz; owner: Aaron Schulz):
[mediawiki/core@master] Make DeferredUpdates avoid running during LBFactory::commitMasterChanges

https://gerrit.wikimedia.org/r/430532

Change 430532 had a related patch set uploaded (by Aaron Schulz; owner: Aaron Schulz):
[mediawiki/core@master] Make DeferredUpdates avoid running during LBFactory::commitMasterChanges

https://gerrit.wikimedia.org/r/430532

Change 430532 merged by jenkins-bot:
[mediawiki/core@master] Make DeferredUpdates avoid running during LBFactory::commitMasterChanges

https://gerrit.wikimedia.org/r/430532

Mentioned in SAL (#wikimedia-operations) [2018-05-03T22:52:27Z] <aaron@tin> Finished scap: Deploy db9acea7eb1c717104691857d1b3ce73c2e18847 (bug T193668) (duration: 103m 50s)

Change 430273 had a related patch set uploaded (by Aaron Schulz; owner: Aaron Schulz):
[mediawiki/core@master] Use AutoCommitUpdate in LinksUpdate::doUpdate

https://gerrit.wikimedia.org/r/430273

Change 430273 merged by jenkins-bot:
[mediawiki/core@master] Use AutoCommitUpdate in LinksUpdate::doUpdate

https://gerrit.wikimedia.org/r/430273

Change 431279 had a related patch set uploaded (by Aaron Schulz; owner: Aaron Schulz):
[mediawiki/core@master] rdbms: fix finalization stage errors in LBFactory::commitMasterChanges

https://gerrit.wikimedia.org/r/431279

greg lowered the priority of this task from Unbreak Now! to High.May 7 2018, 4:14 PM
greg added a subscriber: greg.

No longer UBN! as this was rollback. Removing as blocker (obviously as we're done with wmf.2 now).

Change 431279 merged by jenkins-bot:
[mediawiki/core@master] rdbms: fix finalization stage errors in LBFactory::commitMasterChanges

https://gerrit.wikimedia.org/r/431279

What's the plan now? Can this be closed?

aaron closed this task as Resolved.Jun 14 2018, 7:18 PM
aaron claimed this task.
Vvjjkkii renamed this task from Transaction should be in the callback stage (not 'cursory') to nrdaaaaaaa.Jul 1 2018, 1:12 AM
Vvjjkkii reopened this task as Open.
Vvjjkkii removed aaron as the assignee of this task.
Vvjjkkii updated the task description. (Show Details)
Vvjjkkii removed subscribers: gerritbot, Aklapper.
CommunityTechBot renamed this task from nrdaaaaaaa to Transaction should be in the callback stage (not 'cursory').Jul 2 2018, 5:21 AM
CommunityTechBot closed this task as Resolved.
CommunityTechBot assigned this task to aaron.
CommunityTechBot updated the task description. (Show Details)
CommunityTechBot added subscribers: gerritbot, Aklapper.
mmodell changed the subtype of this task from "Task" to "Production Error".Aug 28 2019, 11:09 PM