"Could not enqueue jobs" error in a lot of places
Closed, ResolvedPublicBUG REPORT
Actions

Assigned To

Authored By

	dom_walden
	Mar 2 2022, 3:47 PM

Description

What is the problem?

When trying to delete a page on beta I got the error:

[Yh@QpJ78nPeWoDJ-nWagkQAAAJg] /w/index.php?title=Conflict-title-0.20908977166972087-I%C3%B1t%C3%ABrn%C3%A2ti%C3%B4n%C3%A0liz%C3%A6ti%C3%B8n&action=delete JobQueueError: Could not enqueue jobs

Backtrace:

from /srv/mediawiki/php-master/extensions/EventBus/includes/Adapters/JobQueue/JobQueueEventBus.php(133)
#0 /srv/mediawiki/php-master/includes/jobqueue/JobQueue.php(376): MediaWiki\Extension\EventBus\Adapters\JobQueue\JobQueueEventBus->doBatchPush(array, integer)
#1 /srv/mediawiki/php-master/includes/jobqueue/JobQueue.php(348): JobQueue->batchPush(array, integer)
#2 /srv/mediawiki/php-master/includes/jobqueue/JobQueueGroup.php(194): JobQueue->push(array)
#3 /srv/mediawiki/php-master/extensions/CirrusSearch/includes/Hooks.php(349): JobQueueGroup->push(array)
#4 /srv/mediawiki/php-master/includes/HookContainer/HookContainer.php(338): CirrusSearch\Hooks::onArticleDeleteComplete(WikiPage, User, string, integer, WikitextContent, ManualLogEntry, integer)
#5 /srv/mediawiki/php-master/includes/HookContainer/HookContainer.php(137): MediaWiki\HookContainer\HookContainer->callLegacyHook(string, array, array, array)
#6 /srv/mediawiki/php-master/includes/HookContainer/HookRunner.php(719): MediaWiki\HookContainer\HookContainer->run(string, array)
#7 /srv/mediawiki/php-master/includes/page/DeletePage.php(697): MediaWiki\HookContainer\HookRunner->onArticleDeleteComplete(WikiPage, User, string, integer, WikitextContent, ManualLogEntry, integer)
#8 /srv/mediawiki/php-master/includes/page/DeletePage.php(479): MediaWiki\Page\DeletePage->deleteInternal(WikiPage, string, string)
#9 /srv/mediawiki/php-master/includes/page/DeletePage.php(381): MediaWiki\Page\DeletePage->deleteUnsafe(string)
#10 /srv/mediawiki/php-master/includes/actions/DeleteAction.php(144): MediaWiki\Page\DeletePage->deleteIfAllowed(string)
#11 /srv/mediawiki/php-master/includes/actions/DeleteAction.php(99): DeleteAction->tempDelete()
#12 /srv/mediawiki/php-master/includes/MediaWiki.php(544): DeleteAction->show()
#13 /srv/mediawiki/php-master/includes/MediaWiki.php(321): MediaWiki->performAction(Article, Title)
#14 /srv/mediawiki/php-master/includes/MediaWiki.php(910): MediaWiki->performRequest()
#15 /srv/mediawiki/php-master/includes/MediaWiki.php(564): MediaWiki->main()
#16 /srv/mediawiki/php-master/index.php(53): MediaWiki->run()
#17 /srv/mediawiki/php-master/index.php(46): wfIndexMain()
#18 /srv/mediawiki/w/index.php(3): require(string)
#19 {main}

Both via API and UI.

This started happening roughly at 15:40UTC.

EDIT: Raising this priority as it is affecting a lot of things, not just deletion:

Steps to reproduce problem

Login as admin to https://en.wikipedia.beta.wmflabs.org
Go to https://en.wikipedia.beta.wmflabs.org/w/index.php?title=0.4014569430451824&action=delete
Click "Delete page"

Expected behavior: Page is deleted.
Observed behavior: Above error.

Environment

Wiki(s): https://en.wikipedia.beta.wmflabs.org MediaWiki 1.38.0-alpha (e16f857) 15:12, 2 March 2022. CirrusSearch 6.5.4 (3461774) 21:37, 28 February 2022. EventBus 0.4.0 (65f7536) 19:40, 26 February 2022.

Related Objects

Mentioned Here: T302699: Beta cluster down: Error: 502, Next Hop Connection Failed (Feb 2022)
rEEVB65f7536e2621: Use new namespace for MediaWiki\Revision\RevisionSlots
rECIR346177486f65: ext.cirrus.serp: Use "wikipage.content" instead of mw.util.$content
rMWe16f857eaea0: Merge "linker: Start of LinkTargetStore"

Event Timeline

dom_walden created this task.Mar 2 2022, 3:47 PM

Restricted Application added projects: Data-Engineering, Discovery-Search. · View Herald TranscriptMar 2 2022, 3:47 PM

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

dom_walden updated the task description. (Show Details)Mar 2 2022, 3:47 PM

dom_walden renamed this task from "Could not enqueue jobs" when trying to delete a page to "Could not enqueue jobs" error in a lot of places.Mar 2 2022, 4:45 PM

dom_walden triaged this task as Unbreak Now! priority.

dom_walden updated the task description. (Show Details)

dom_walden added a project: Platform Engineering.Mar 2 2022, 4:53 PM

We are investigating whether this was caused by https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/756635.

Reverting to see if that fixes this https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/767101.

In T302887#7748065, @dom_walden wrote:

Reverting to see if that fixes this https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/767101.

This has appeared to fix the problem.

We pushed a new config patch: https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/767558

dom_walden removed projects: Platform Engineering, Discovery-Search, Data-Engineering, Event-Platform, MediaWiki-Page-deletion, CirrusSearch.Mar 2 2022, 6:02 PM

@dom_walden Is this OK to close now?

In T302887#7753491, @Tchanders wrote:

@dom_walden Is this OK to close now?

Beta has been working fine since we deployed the fixed config (apart from T302699 but that has nothing to do with us).

I will close.

• Niharika moved this task from Backlog to Closed on the IP Info board.Aug 8 2022, 7:58 PM

"Could not enqueue jobs" error in a lot of placesClosed, ResolvedPublicBUG REPORTActions