Page MenuHomePhabricator
Create Task
Maniphest T371433

JobQueueError: Could not enqueue jobs
Closed, ResolvedPublicPRODUCTION ERROR
Actions

Description

Error
normalized_message
[{reqId}] {exception_url}   JobQueueError: Could not enqueue jobs
exception.trace
from /srv/mediawiki/php-1.43.0-wmf.15/extensions/EventBus/includes/Adapters/JobQueue/JobQueueEventBus.php(134)
#0 /srv/mediawiki/php-1.43.0-wmf.15/includes/jobqueue/JobQueue.php(380): MediaWiki\Extension\EventBus\Adapters\JobQueue\JobQueueEventBus->doBatchPush(array, integer)
#1 /srv/mediawiki/php-1.43.0-wmf.15/includes/jobqueue/JobQueue.php(352): JobQueue->batchPush(array, integer)
#2 /srv/mediawiki/php-1.43.0-wmf.15/includes/jobqueue/JobQueueGroup.php(157): JobQueue->push(array)
#3 /srv/mediawiki/php-1.43.0-wmf.15/extensions/Echo/includes/Controller/NotificationController.php(226): JobQueueGroup->push(array)
#4 /srv/mediawiki/php-1.43.0-wmf.15/extensions/Echo/includes/Controller/NotificationController.php(173): MediaWiki\Extension\Notifications\Controller\NotificationController::enqueueDeleteJob(array, MediaWiki\Extension\Notifications\Model\Event)
#5 /srv/mediawiki/php-1.43.0-wmf.15/extensions/Echo/includes/Model/Event.php(207): MediaWiki\Extension\Notifications\Controller\NotificationController::notify(MediaWiki\Extension\Notifications\Model\Event, boolean)
#6 /srv/mediawiki/php-1.43.0-wmf.15/extensions/LoginNotify/includes/LoginNotify.php(1096): MediaWiki\Extension\Notifications\Model\Event::create(array)
#7 /srv/mediawiki/php-1.43.0-wmf.15/extensions/LoginNotify/includes/LoginNotify.php(1215): LoginNotify\LoginNotify->sendNotice(MediaWiki\User\User, string)
#8 /srv/mediawiki/php-1.43.0-wmf.15/extensions/LoginNotify/includes/Hooks.php(71): LoginNotify\LoginNotify->sendSuccessNotice(MediaWiki\User\User)
#9 /srv/mediawiki/php-1.43.0-wmf.15/extensions/LoginNotify/includes/Hooks.php(51): LoginNotify\Hooks::doSuccessfulLogin(MediaWiki\User\User)
#10 /srv/mediawiki/php-1.43.0-wmf.15/includes/HookContainer/HookContainer.php(159): LoginNotify\Hooks->onAuthManagerLoginAuthenticateAudit(MediaWiki\Auth\AuthenticationResponse, MediaWiki\User\User, string, array)
#11 /srv/mediawiki/php-1.43.0-wmf.15/includes/HookContainer/HookRunner.php(894): MediaWiki\HookContainer\HookContainer->run(string, array)
#12 /srv/mediawiki/php-1.43.0-wmf.15/includes/auth/AuthManager.php(853): MediaWiki\HookContainer\HookRunner->onAuthManagerLoginAuthenticateAudit(MediaWiki\Auth\AuthenticationResponse, MediaWiki\User\User, string, array)
#13 /srv/mediawiki/php-1.43.0-wmf.15/includes/specialpage/AuthManagerSpecialPage.php(393): MediaWiki\Auth\AuthManager->continueAuthentication(array)
#14 /srv/mediawiki/php-1.43.0-wmf.15/includes/specialpage/AuthManagerSpecialPage.php(525): MediaWiki\SpecialPage\AuthManagerSpecialPage->performAuthenticationStep(string, array)
#15 [internal function]: MediaWiki\SpecialPage\AuthManagerSpecialPage->handleFormSubmit(array, MediaWiki\HTMLForm\CodexHTMLForm)
#16 /srv/mediawiki/php-1.43.0-wmf.15/includes/htmlform/HTMLForm.php(793): call_user_func(array, array, MediaWiki\HTMLForm\CodexHTMLForm)
#17 /srv/mediawiki/php-1.43.0-wmf.15/includes/specialpage/AuthManagerSpecialPage.php(456): MediaWiki\HTMLForm\HTMLForm->trySubmit()
#18 /srv/mediawiki/php-1.43.0-wmf.15/includes/specialpage/LoginSignupSpecialPage.php(383): MediaWiki\SpecialPage\AuthManagerSpecialPage->trySubmit()
#19 /srv/mediawiki/php-1.43.0-wmf.15/includes/specialpage/SpecialPage.php(719): MediaWiki\SpecialPage\LoginSignupSpecialPage->execute(NULL)
#20 /srv/mediawiki/php-1.43.0-wmf.15/includes/specialpage/SpecialPageFactory.php(1693): MediaWiki\SpecialPage\SpecialPage->run(NULL)
#21 /srv/mediawiki/php-1.43.0-wmf.15/includes/actions/ActionEntryPoint.php(502): MediaWiki\SpecialPage\SpecialPageFactory->executePath(string, MediaWiki\Context\RequestContext)
#22 /srv/mediawiki/php-1.43.0-wmf.15/includes/actions/ActionEntryPoint.php(145): MediaWiki\Actions\ActionEntryPoint->performRequest()
#23 /srv/mediawiki/php-1.43.0-wmf.15/includes/MediaWikiEntryPoint.php(200): MediaWiki\Actions\ActionEntryPoint->execute()
#24 /srv/mediawiki/php-1.43.0-wmf.15/index.php(58): MediaWiki\MediaWikiEntryPoint->run()
#25 /srv/mediawiki/w/index.php(3): require(string)
#26 {main}
Impact

Occured when I tried to log in on checkuserwiki and prevented login.

Notes

Seems to only impact private wikis, e.g., checkuserwiki, officewiki, stewardswiki, sysop_plwiki.

Details

Request URL
https://checkuser.wikimedia.org/w/index.php?returnto=*&title=*
Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
EventStreamConfig - fix for private wiki streamsOttomataoperations/mediawiki-configmaster+24 -0
Revert "Produce a limited set of event streams on private wikis"Máté Szabóoperations/mediawiki-configmaster+1 -63
EventStreamConfig: Re-enable mediawiki_eventbus on private wikisUrbanecmoperations/mediawiki-configmaster+0 -7
Customize query in gerrit

Related Objects

Event Timeline

RAdimer-WMF created this task.Jul 30 2024, 8:06 PM
Restricted Application added a project: Data-Engineering. · View Herald TranscriptJul 30 2024, 8:06 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Ladsgroup subscribed.Jul 31 2024, 10:33 AM
Dreamy_Jazz triaged this task as Unbreak Now! priority.Jul 31 2024, 10:40 AM
Dreamy_Jazz subscribed.

Prevents login for me to. Impacts abuse-mitigation that involves looking at the CheckUser wiki data.

gerritbot added a comment.Jul 31 2024, 10:58 AM

Change #1058582 had a related patch set uploaded (by Urbanecm; author: Urbanecm):

[operations/mediawiki-config@master] EventStreamConfig: Re-enable mediawiki_eventbus on private wikis

https://gerrit.wikimedia.org/r/1058582

gerritbot added a project: Patch-For-Review.Jul 31 2024, 10:58 AM

Change #1058582 merged by jenkins-bot:

[operations/mediawiki-config@master] EventStreamConfig: Re-enable mediawiki_eventbus on private wikis

https://gerrit.wikimedia.org/r/1058582

gerritbot added a comment.Jul 31 2024, 11:00 AM

Change #1058583 had a related patch set uploaded (by Máté Szabó; author: Máté Szabó):

[operations/mediawiki-config@master] Revert "Produce a limited set of event streams on private wikis"

https://gerrit.wikimedia.org/r/1058583

• mszabo merged a task: T371471: Could not enqueue jobs for stream {stream}.Jul 31 2024, 11:01 AM
• mszabo subscribed.
gerritbot added a comment.Jul 31 2024, 11:02 AM

Change #1058583 abandoned by Máté Szabó:

[operations/mediawiki-config@master] Revert "Produce a limited set of event streams on private wikis"

Reason:

Superseded by https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/1058582

https://gerrit.wikimedia.org/r/1058583

Stashbot added a comment.Jul 31 2024, 11:03 AM

Mentioned in SAL (#wikimedia-operations) [2024-07-31T11:03:39Z] <urbanecm@deploy1003> Started scap sync-world: Backport for [[gerrit:1058582|EventStreamConfig: Re-enable mediawiki_eventbus on private wikis (T371433)]]

Stashbot added a comment.Jul 31 2024, 11:05 AM

Mentioned in SAL (#wikimedia-operations) [2024-07-31T11:05:51Z] <urbanecm@deploy1003> urbanecm: Backport for [[gerrit:1058582|EventStreamConfig: Re-enable mediawiki_eventbus on private wikis (T371433)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)

Ladsgroup added subscribers: Gehel, Ottomata, EBernhardson.Jul 31 2024, 11:07 AM
Stashbot added a comment.Jul 31 2024, 11:11 AM

Mentioned in SAL (#wikimedia-operations) [2024-07-31T11:11:42Z] <urbanecm@deploy1003> Finished scap: Backport for [[gerrit:1058582|EventStreamConfig: Re-enable mediawiki_eventbus on private wikis (T371433)]] (duration: 08m 02s)

Urbanecm_WMF subscribed.Jul 31 2024, 11:23 AM

Okay, jobs should now be back. @Dreamy_Jazz can now login to checkuserwiki with no issues, and as far as I can see, other jobs on private wikis do their job as well.

Urbanecm_WMF mentioned this in T346046: [Search Update Pipeline] Source streams for private wikis.Jul 31 2024, 11:23 AM
Maintenance_bot removed a project: Patch-For-Review.Jul 31 2024, 11:30 AM
Urbanecm_WMF closed this task as Resolved.Jul 31 2024, 12:55 PM
Urbanecm_WMF claimed this task.
bking subscribed.Jul 31 2024, 1:04 PM
gerritbot added a comment.Jul 31 2024, 1:33 PM

Change #1058603 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/mediawiki-config@master] EventStreamConfig - fix for private wiki streams

https://gerrit.wikimedia.org/r/1058603

gerritbot added a project: Patch-For-Review.Jul 31 2024, 1:33 PM
Ottomata added a comment.Jul 31 2024, 1:36 PM

Thank you okay!

So EventStreamConfig: Re-enable mediawiki_eventbus on private wikis (1058582) will have reenabled jobs, but also has started producing TYPE_EVENTs from private wikis into public streams.

https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/1058603 will hopefully fix.

I'm sorry this is so confusing. T370524: Simplify EventBus code by deprecating EventBusFactory would have simplified but we decided we didn't have time for the refactor at the moment :(

gerritbot added a comment.Jul 31 2024, 1:39 PM

Change #1058603 merged by jenkins-bot:

[operations/mediawiki-config@master] EventStreamConfig - fix for private wiki streams

https://gerrit.wikimedia.org/r/1058603

Stashbot added a comment.Jul 31 2024, 1:42 PM

Mentioned in SAL (#wikimedia-operations) [2024-07-31T13:42:16Z] <logmsgbot> lucaswerkmeister-wmde@deploy1003 Started scap sync-world: Backport for [[gerrit:1058603|EventStreamConfig - fix for private wiki streams (T346046 T371433)]]

Stashbot added a comment.Jul 31 2024, 1:44 PM

Mentioned in SAL (#wikimedia-operations) [2024-07-31T13:44:23Z] <logmsgbot> lucaswerkmeister-wmde@deploy1003 lucaswerkmeister-wmde, otto: Backport for [[gerrit:1058603|EventStreamConfig - fix for private wiki streams (T346046 T371433)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)

Stashbot added a comment.Jul 31 2024, 1:53 PM

Mentioned in SAL (#wikimedia-operations) [2024-07-31T13:53:48Z] <logmsgbot> lucaswerkmeister-wmde@deploy1003 Finished scap: Backport for [[gerrit:1058603|EventStreamConfig - fix for private wiki streams (T346046 T371433)]] (duration: 11m 31s)

Ottomata added a comment.Jul 31 2024, 2:12 PM

EventStreamConfig - fix for private wiki streams (1058603) is deployed. Please let us know if anything is broken by it!

Maintenance_bot removed a project: Patch-For-Review.Jul 31 2024, 2:36 PM
dr0ptp4kt subscribed.Jul 31 2024, 5:28 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits