JobQueueError: Could not enqueue jobs
Closed, ResolvedPublicPRODUCTION ERROR
Actions

Assigned To

Authored By

	RAdimer-WMF
	Jul 30 2024, 8:06 PM

Description

Error

mwversion: 1.43.0-wmf.15
reqId: c5cc8b40-e70c-456b-aa8d-24430092e3d0
Find reqId in Logstash

normalized_message

[{reqId}] {exception_url}   JobQueueError: Could not enqueue jobs

exception.trace

from /srv/mediawiki/php-1.43.0-wmf.15/extensions/EventBus/includes/Adapters/JobQueue/JobQueueEventBus.php(134)
#0 /srv/mediawiki/php-1.43.0-wmf.15/includes/jobqueue/JobQueue.php(380): MediaWiki\Extension\EventBus\Adapters\JobQueue\JobQueueEventBus->doBatchPush(array, integer)
#1 /srv/mediawiki/php-1.43.0-wmf.15/includes/jobqueue/JobQueue.php(352): JobQueue->batchPush(array, integer)
#2 /srv/mediawiki/php-1.43.0-wmf.15/includes/jobqueue/JobQueueGroup.php(157): JobQueue->push(array)
#3 /srv/mediawiki/php-1.43.0-wmf.15/extensions/Echo/includes/Controller/NotificationController.php(226): JobQueueGroup->push(array)
#4 /srv/mediawiki/php-1.43.0-wmf.15/extensions/Echo/includes/Controller/NotificationController.php(173): MediaWiki\Extension\Notifications\Controller\NotificationController::enqueueDeleteJob(array, MediaWiki\Extension\Notifications\Model\Event)
#5 /srv/mediawiki/php-1.43.0-wmf.15/extensions/Echo/includes/Model/Event.php(207): MediaWiki\Extension\Notifications\Controller\NotificationController::notify(MediaWiki\Extension\Notifications\Model\Event, boolean)
#6 /srv/mediawiki/php-1.43.0-wmf.15/extensions/LoginNotify/includes/LoginNotify.php(1096): MediaWiki\Extension\Notifications\Model\Event::create(array)
#7 /srv/mediawiki/php-1.43.0-wmf.15/extensions/LoginNotify/includes/LoginNotify.php(1215): LoginNotify\LoginNotify->sendNotice(MediaWiki\User\User, string)
#8 /srv/mediawiki/php-1.43.0-wmf.15/extensions/LoginNotify/includes/Hooks.php(71): LoginNotify\LoginNotify->sendSuccessNotice(MediaWiki\User\User)
#9 /srv/mediawiki/php-1.43.0-wmf.15/extensions/LoginNotify/includes/Hooks.php(51): LoginNotify\Hooks::doSuccessfulLogin(MediaWiki\User\User)
#10 /srv/mediawiki/php-1.43.0-wmf.15/includes/HookContainer/HookContainer.php(159): LoginNotify\Hooks->onAuthManagerLoginAuthenticateAudit(MediaWiki\Auth\AuthenticationResponse, MediaWiki\User\User, string, array)
#11 /srv/mediawiki/php-1.43.0-wmf.15/includes/HookContainer/HookRunner.php(894): MediaWiki\HookContainer\HookContainer->run(string, array)
#12 /srv/mediawiki/php-1.43.0-wmf.15/includes/auth/AuthManager.php(853): MediaWiki\HookContainer\HookRunner->onAuthManagerLoginAuthenticateAudit(MediaWiki\Auth\AuthenticationResponse, MediaWiki\User\User, string, array)
#13 /srv/mediawiki/php-1.43.0-wmf.15/includes/specialpage/AuthManagerSpecialPage.php(393): MediaWiki\Auth\AuthManager->continueAuthentication(array)
#14 /srv/mediawiki/php-1.43.0-wmf.15/includes/specialpage/AuthManagerSpecialPage.php(525): MediaWiki\SpecialPage\AuthManagerSpecialPage->performAuthenticationStep(string, array)
#15 [internal function]: MediaWiki\SpecialPage\AuthManagerSpecialPage->handleFormSubmit(array, MediaWiki\HTMLForm\CodexHTMLForm)
#16 /srv/mediawiki/php-1.43.0-wmf.15/includes/htmlform/HTMLForm.php(793): call_user_func(array, array, MediaWiki\HTMLForm\CodexHTMLForm)
#17 /srv/mediawiki/php-1.43.0-wmf.15/includes/specialpage/AuthManagerSpecialPage.php(456): MediaWiki\HTMLForm\HTMLForm->trySubmit()
#18 /srv/mediawiki/php-1.43.0-wmf.15/includes/specialpage/LoginSignupSpecialPage.php(383): MediaWiki\SpecialPage\AuthManagerSpecialPage->trySubmit()
#19 /srv/mediawiki/php-1.43.0-wmf.15/includes/specialpage/SpecialPage.php(719): MediaWiki\SpecialPage\LoginSignupSpecialPage->execute(NULL)
#20 /srv/mediawiki/php-1.43.0-wmf.15/includes/specialpage/SpecialPageFactory.php(1693): MediaWiki\SpecialPage\SpecialPage->run(NULL)
#21 /srv/mediawiki/php-1.43.0-wmf.15/includes/actions/ActionEntryPoint.php(502): MediaWiki\SpecialPage\SpecialPageFactory->executePath(string, MediaWiki\Context\RequestContext)
#22 /srv/mediawiki/php-1.43.0-wmf.15/includes/actions/ActionEntryPoint.php(145): MediaWiki\Actions\ActionEntryPoint->performRequest()
#23 /srv/mediawiki/php-1.43.0-wmf.15/includes/MediaWikiEntryPoint.php(200): MediaWiki\Actions\ActionEntryPoint->execute()
#24 /srv/mediawiki/php-1.43.0-wmf.15/index.php(58): MediaWiki\MediaWikiEntryPoint->run()
#25 /srv/mediawiki/w/index.php(3): require(string)
#26 {main}

Impact

Occured when I tried to log in on checkuserwiki and prevented login.

Notes

Seems to only impact private wikis, e.g., checkuserwiki, officewiki, stewardswiki, sysop_plwiki.

Details

Request URL: https://checkuser.wikimedia.org/w/index.php?returnto=*&title=*

Subject	Repo	Branch	Lines +/-
EventStreamConfig - fix for private wiki streams	operations/mediawiki-config	master	+24 -0
Revert "Produce a limited set of event streams on private wikis"	operations/mediawiki-config	master	+1 -63
EventStreamConfig: Re-enable mediawiki_eventbus on private wikis	operations/mediawiki-config	master	+0 -7

Customize query in gerrit

Related Objects

Mentioned In: T346046: [Search Update Pipeline] Source streams for private wikis
Mentioned Here: T346046: [Search Update Pipeline] Source streams for private wikis
T370524: Simplify EventBus code by deprecating EventBusFactory

Event Timeline

RAdimer-WMF created this task.Jul 30 2024, 8:06 PM

Restricted Application added a project: Data-Engineering. · View Herald TranscriptJul 30 2024, 8:06 PM

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

Ladsgroup subscribed.Jul 31 2024, 10:33 AM

Prevents login for me to. Impacts abuse-mitigation that involves looking at the CheckUser wiki data.

Change #1058582 had a related patch set uploaded (by Urbanecm; author: Urbanecm):

[operations/mediawiki-config@master] EventStreamConfig: Re-enable mediawiki_eventbus on private wikis

https://gerrit.wikimedia.org/r/1058582

Change #1058582 merged by jenkins-bot:

[operations/mediawiki-config@master] EventStreamConfig: Re-enable mediawiki_eventbus on private wikis

https://gerrit.wikimedia.org/r/1058582

Change #1058583 had a related patch set uploaded (by Máté Szabó; author: Máté Szabó):

[operations/mediawiki-config@master] Revert "Produce a limited set of event streams on private wikis"

https://gerrit.wikimedia.org/r/1058583

mszabo merged a task: T371471: Could not enqueue jobs for stream {stream}.Jul 31 2024, 11:01 AM

mszabo subscribed.

Change #1058583 abandoned by Máté Szabó:

[operations/mediawiki-config@master] Revert "Produce a limited set of event streams on private wikis"

Reason:

Superseded by https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/1058582

https://gerrit.wikimedia.org/r/1058583

Mentioned in SAL (#wikimedia-operations) [2024-07-31T11:03:39Z] <urbanecm@deploy1003> Started scap sync-world: Backport for [[gerrit:1058582|EventStreamConfig: Re-enable mediawiki_eventbus on private wikis (T371433)]]

Mentioned in SAL (#wikimedia-operations) [2024-07-31T11:05:51Z] <urbanecm@deploy1003> urbanecm: Backport for [[gerrit:1058582|EventStreamConfig: Re-enable mediawiki_eventbus on private wikis (T371433)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)

Ladsgroup added subscribers: Gehel, Ottomata, EBernhardson.Jul 31 2024, 11:07 AM

Mentioned in SAL (#wikimedia-operations) [2024-07-31T11:11:42Z] <urbanecm@deploy1003> Finished scap: Backport for [[gerrit:1058582|EventStreamConfig: Re-enable mediawiki_eventbus on private wikis (T371433)]] (duration: 08m 02s)

Okay, jobs should now be back. @Dreamy_Jazz can now login to checkuserwiki with no issues, and as far as I can see, other jobs on private wikis do their job as well.

Urbanecm_WMF mentioned this in T346046: [Search Update Pipeline] Source streams for private wikis.Jul 31 2024, 11:23 AM

Maintenance_bot removed a project: Patch-For-Review.Jul 31 2024, 11:30 AM

Urbanecm_WMF closed this task as Resolved.Jul 31 2024, 12:55 PM

Urbanecm_WMF claimed this task.

bking subscribed.Jul 31 2024, 1:04 PM

Change #1058603 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/mediawiki-config@master] EventStreamConfig - fix for private wiki streams

https://gerrit.wikimedia.org/r/1058603

gerritbot added a project: Patch-For-Review.Jul 31 2024, 1:33 PM

Thank you okay!

So EventStreamConfig: Re-enable mediawiki_eventbus on private wikis (1058582) will have reenabled jobs, but also has started producing TYPE_EVENTs from private wikis into public streams.

https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/1058603 will hopefully fix.

I'm sorry this is so confusing. T370524: Simplify EventBus code by deprecating EventBusFactory would have simplified but we decided we didn't have time for the refactor at the moment :(

Change #1058603 merged by jenkins-bot:

[operations/mediawiki-config@master] EventStreamConfig - fix for private wiki streams

https://gerrit.wikimedia.org/r/1058603

Mentioned in SAL (#wikimedia-operations) [2024-07-31T13:42:16Z] <logmsgbot> lucaswerkmeister-wmde@deploy1003 Started scap sync-world: Backport for [[gerrit:1058603|EventStreamConfig - fix for private wiki streams (T346046 T371433)]]

Mentioned in SAL (#wikimedia-operations) [2024-07-31T13:44:23Z] <logmsgbot> lucaswerkmeister-wmde@deploy1003 lucaswerkmeister-wmde, otto: Backport for [[gerrit:1058603|EventStreamConfig - fix for private wiki streams (T346046 T371433)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)

Mentioned in SAL (#wikimedia-operations) [2024-07-31T13:53:48Z] <logmsgbot> lucaswerkmeister-wmde@deploy1003 Finished scap: Backport for [[gerrit:1058603|EventStreamConfig - fix for private wiki streams (T346046 T371433)]] (duration: 11m 31s)

EventStreamConfig - fix for private wiki streams (1058603) is deployed. Please let us know if anything is broken by it!

Maintenance_bot removed a project: Patch-For-Review.Jul 31 2024, 2:36 PM

dr0ptp4kt subscribed.Jul 31 2024, 5:28 PM

JobQueueError: Could not enqueue jobsClosed, ResolvedPublicPRODUCTION ERRORActions