Page MenuHomePhabricator
Create Task
Maniphest T268840

Infinite loop in DeferredUpdates::tryOpportunisticExecute
Closed, ResolvedPublic
Actions

Description

This is split from T252764: Automatic renaming by Fuzzybot broken to focus on the underlying issue. It is likely that our rename jobs trigger a rare code path due to them causing a lot of deferred updates.

The trace is:

  • require_once
  • RunJobs->execute
  • JobRunner->run
  • JobRunner->executeJob
  • JobRunner->doExecuteJob
  • MessageUpdateJob->run
  • MessageUpdateJob->handleRename
  • MovePage->move
  • MovePage->moveUnsafe
  • MovePage->moveToInternal
  • ManualLogEntry->publish
  • DeferredUpdates::addCallableUpdate
  • DeferredUpdates::addUpdate
  • DeferredUpdates::tryOpportunisticExecute [loop starts]
  • DeferredUpdates::enqueueUpdates
  • JobQueueGroup->push [1]
  • JobQueue->push
  • JobQueue->batchPush
  • JobQueueDB->doBatchPush
  • Wikimedia\Rdbms\DBConnRef->onTransactionPreCommitOrIdle
  • Wikimedia\Rdbms\DBConnRef->__call
  • Wikimedia\Rdbms\Database->onTransactionPreCommitOrIdle
  • Wikimedia\Rdbms\Database->endAtomic
  • Wikimedia\Rdbms\Database->commit
  • Wikimedia\Rdbms\Database->runTransactionListenerCallbacks
  • Maintenance::{closure}
  • DeferredUpdates::tryOpportunisticExecute [loop restarts]

The job that is getting pushed in [1] is RefreshSecondaryDataUpdate

The jobs in the queue are as follows:
self::$preSendUpdates:

  • MessageCacheUpdate
  • CdnCacheUpdate

self::$postSendUpdates:

  • MWCallableUpdate
  • MWCallableUpdate
  • MWCallableUpdate
  • MWCallableUpdate
  • RefreshSecondaryDataUpdate

Do note that we are hitting the self::BIG_QUEUE_SIZEcondition in rMW includes/deferred/DeferredUpdates.php:495-500

The issue is, I believe, that enqueueUpdates is not safe for recursive entrance, as it does not remove items from the queue before inserting them to the job queue (if they are queueable), which causes another opportunity to execute deferred updates.

Details

SubjectRepoBranchLines +/-
deferred: make DeferredUpdates::doUpdates() recursion more uniformmediawiki/coremaster+690 -335
Customize query in gerrit

Related Objects
Search...

Event Timeline

• Nikerabbit created this task.Nov 26 2020, 4:02 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 26 2020, 4:02 PM
• Nikerabbit added a parent task: T252764: Automatic renaming by Fuzzybot broken.Nov 26 2020, 4:03 PM
• Nikerabbit mentioned this in T252764: Automatic renaming by Fuzzybot broken.
• Nikerabbit added a comment.Nov 26 2020, 4:07 PM

I think this is the underlying cause for T260812: Moving of translation page fails with OOM exception (database job queue backend) as well. I think WMF environment is not affected since it does not use the database backend for the job queue and hence does not trigger this loop.

• Clarakosi triaged this task as Low priority.Dec 1 2020, 9:09 PM
• Clarakosi edited projects, added Platform Team Workboards (Clinic Duty Team); removed Platform Engineering.
• Clarakosi moved this task from Inbox to Later on the Platform Team Workboards (Clinic Duty Team) board.
• Nikerabbit moved this task from Backlog to Mitigated on the affects-translatewiki.net board.Dec 7 2020, 7:41 AM

Commenting out these two lines locally makes the issue go away. I'm not sure what are the drawbacks of a having a big queue size, so I'm leaving it to you to figure out what is an appropriate fix.

		if ( self::pendingUpdatesCount() >= self::BIG_QUEUE_SIZE ) {
				// If we cannot run the updates with outer transaction context, try to
				// at least enqueue all the updates that support queueing to job queue
				// self::$preSendUpdates = self::enqueueUpdates( self::$preSendUpdates );
				// self::$postSendUpdates = self::enqueueUpdates( self::$postSendUpdates );
			}
• Nikerabbit added a parent task: T260812: Moving of translation page fails with OOM exception (database job queue backend).Dec 10 2020, 12:20 PM
• Nikerabbit mentioned this in T260812: Moving of translation page fails with OOM exception (database job queue backend).Dec 14 2020, 3:51 PM
• Nikerabbit added subscribers: aaron, Krinkle.Dec 21 2020, 7:06 AM

Tagging @aaron and @Krinkle who removed the recursion guard in rMWc1ebcb0ee24d: Remove $recursionGuard var from tryOpportunisticExecute(). Could you have a look?

This ticket blocks two high priority tickets for the Language team.

CCicalese_WMF edited projects, added Platform Engineering; removed Platform Team Workboards (Clinic Duty Team).Jan 5 2021, 8:40 PM
• holger.knust raised the priority of this task from Low to Medium.Jan 5 2021, 9:26 PM
• holger.knust edited projects, added Platform Team Workboards (Clinic Duty Team); removed Platform Engineering.
• holger.knust moved this task from Later to Ready (WIP:5) on the Platform Team Workboards (Clinic Duty Team) board.
gerritbot added a comment.Jan 8 2021, 7:12 AM

Change 596082 had a related patch set uploaded (by Aaron Schulz; owner: Aaron Schulz):
[mediawiki/core@master] deferred: make DeferredUpdates::doUpdates() recursion more uniform

https://gerrit.wikimedia.org/r/596082

gerritbot added a project: Patch-For-Review.Jan 8 2021, 7:12 AM
Naike added subscribers: • Rileych, Naike.Jan 11 2021, 8:24 PM

@Rileych is PET still a blocker?

• Rileych lowered the priority of this task from Medium to Low.EditedJan 13 2021, 1:46 AM

@Naike

This task is blocking two tasks (T260812 & T252764) for the Language Team. Can PET support us by code reviewing this patch https://gerrit.wikimedia.org/r/c/mediawiki/core/+/596082/?

We would appreciate assistance ASAP; when can we expect a timeline?

• Rileych raised the priority of this task from Low to High.Jan 13 2021, 1:46 AM
gerritbot added a comment.Jan 15 2021, 4:52 PM

Change 596082 merged by jenkins-bot:
[mediawiki/core@master] deferred: make DeferredUpdates::doUpdates() recursion more uniform

https://gerrit.wikimedia.org/r/596082

ReleaseTaggerBot added a project: MW-1.36-notes (1.36.0-wmf.27; 2021-01-19).Jan 15 2021, 5:00 PM
Maintenance_bot removed a project: Patch-For-Review.Jan 15 2021, 5:11 PM
Buovjaga mentioned this in T20585: Moving many (>100) subpages requires loads of memory; leads to PHP Fatal error.Jan 18 2021, 9:17 AM
• Nikerabbit added a comment.Jan 20 2021, 3:42 PM

Abijeet and I have been doing some testing on this and it looks good so far. We are still planning to do more testing, after which I am planning to mark this ticket as resolved if nothing comes up. Thanks for the help everyone.

• Nikerabbit closed this task as Resolved.Jan 26 2021, 1:27 PM
• Nikerabbit claimed this task.

There has been a couple of small message key renames, but not a large one. I'm marking this resolved, but will reopen in the unlikely case that this issue wasn't fully resolved.

• Nikerabbit mentioned this in T237163: TranslatePageMoveJob segmentation fault.Nov 15 2021, 8:49 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits