JobQueueErrors (Could not enqueue jobs) on Beta Cluster
Closed, ResolvedPublicBUG REPORT
Actions

Assigned To

Authored By

	Quiddity
	Nov 3 2022, 4:08 PM

Description

Steps to replicate the issue:

At a Beta Cluster wiki (tested at EnBeta and at WikifunctionsBeta) in any standard namespace
Visit a history page, or diff page
Click "Thank" for an edit.

What happens?:

Thank action failed (error code: internal_api_error_JobQueueError). Please try again.

What should have happened instead?:

Successfully send a Thank

[Y2Qip-30XgbdpvgTbKuwBgAAAA0] Exception caught: Could not enqueue jobs

JobQueueError at /srv/mediawiki/php-master/extensions/EventBus/includes/Adapters/JobQueue/JobQueueEventBus.php(133) from /srv/mediawiki/php-master/extensions/EventBus/includes/Adapters/JobQueue/JobQueueEventBus.php(133)
#0 /srv/mediawiki/php-master/includes/jobqueue/JobQueue.php(376): MediaWiki\Extension\EventBus\Adapters\JobQueue\JobQueueEventBus->doBatchPush(array, integer)
#1 /srv/mediawiki/php-master/includes/jobqueue/JobQueue.php(348): JobQueue->batchPush(array, integer)
#2 /srv/mediawiki/php-master/includes/jobqueue/JobQueueGroup.php(166): JobQueue->push(array)
#3 /srv/mediawiki/php-master/extensions/Echo/includes/Controller/NotificationController.php(223): JobQueueGroup->push(array)
#4 /srv/mediawiki/php-master/extensions/Echo/includes/Controller/NotificationController.php(172): MediaWiki\Extension\Notifications\Controller\NotificationController::enqueueDeleteJob(array, EchoEvent)
#5 /srv/mediawiki/php-master/extensions/Echo/includes/model/Event.php(193): MediaWiki\Extension\Notifications\Controller\NotificationController::notify(EchoEvent, boolean)
#6 /srv/mediawiki/php-master/extensions/Thanks/includes/ApiCoreThank.php(235): EchoEvent::create(array)
#7 /srv/mediawiki/php-master/extensions/Thanks/includes/ApiCoreThank.php(90): MediaWiki\Extension\Thanks\ApiCoreThank->sendThanks(User, string, integer, string, User, string, Title, boolean)
#8 /srv/mediawiki/php-master/includes/api/ApiMain.php(1902): MediaWiki\Extension\Thanks\ApiCoreThank->execute()
#9 /srv/mediawiki/php-master/includes/api/ApiMain.php(877): ApiMain->executeAction()
#10 /srv/mediawiki/php-master/includes/api/ApiMain.php(848): ApiMain->executeActionWithErrorHandling()
#11 /srv/mediawiki/php-master/api.php(90): ApiMain->execute()
#12 /srv/mediawiki/php-master/api.php(45): wfApiMain()
#13 /srv/mediawiki/w/api.php(3): require(string)#14 {main}

Noting that in T322442: Account creation broken on beta, this is (now?) preventing account creation on the Beta Cluster.

Details

	Project	Branch	Lines +/-	Subject
	mediawiki/extensions/Echo	master	+1 -1	Update job name to match $wgJobClasses key
	mediawiki/extensions/Echo	master	+1 -1	Fix NotificationDeleteJob class name

Customize query in gerrit

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Resolved	Release	brennen	T320515 1.40.0-wmf.10 deployment blockers
		Resolved	BUG REPORT	Zabe	T322342 JobQueueErrors (Could not enqueue jobs) on Beta Cluster

Event Timeline

Quiddity created this task.Nov 3 2022, 4:08 PM

Restricted Application added a project: Growth-Team. · View Herald TranscriptNov 3 2022, 4:08 PM

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

TheresNoTime added a subscriber: TheresNoTime.Nov 3 2022, 8:20 PM

TheresNoTime renamed this task from Error with Thanks on Beta Cluster - internal_api_error_JobQueueError to Exception caught: Could not enqueue jobs when attempting to 'thank' on Beta Cluster.Nov 3 2022, 8:23 PM

TheresNoTime updated the task description. (Show Details)

Seems to happen for many different kinds of jobs, starting on Oct 22 and then vastly increasing in volume on Nov 2.

Restricted Application added a project: Data-Engineering. · View Herald TranscriptNov 5 2022, 6:46 AM

TheresNoTime renamed this task from Exception caught: Could not enqueue jobs when attempting to 'thank' on Beta Cluster to JobQueueErrors (Could not enqueue jobs) on Beta Cluster.Nov 5 2022, 7:21 AM

TheresNoTime merged a task: T322442: Account creation broken on beta.

TheresNoTime updated the task description. (Show Details)

TheresNoTime added a subscriber: vaughnwalters.

On attempting to create an account, I logged this event for req Y2YP1VrG9juoZpSmy0nhNQAAAAM — perhaps of note is the invalid topic mentioned?

{
   "invalid":[
      
   ],
   "error":[
      {
         "status":"error",
         "event":{
            "$schema":"/mediawiki/job/1.0.0",
            "meta":{
               "uri":"https://placeholder.invalid/wiki/Special:Badtitle",
               "request_id":"Y2YP1VrG9juoZpSmy0nhNQAAAAM",
               "id":"42c33faa-52f1-4db9-992d-a991e1156cfc",
               "dt":"2022-11-05T07:25:11Z",
               "domain":"en.wikipedia.beta.wmflabs.org",
               "stream":"mediawiki.job.MediaWiki\\Extension\\Notifications\\Jobs\\NotificationDeleteJob"
            },
            "database":"enwiki",
            "type":"MediaWiki\\Extension\\Notifications\\Jobs\\NotificationDeleteJob",
            "params":{
               "userIds":{
                  "99620":99620
               },
               "namespace":0,
               "title":"Main_Page",
               "requestId":"Y2YP1VrG9juoZpSmy0nhNQAAAAM"
            },
            "mediawiki_signature":"93cdb3cef48c632835127bd925b5f4676bf7113b"
         },
         "context":{
            "message":"invalid topic"
         }
      }
   ]
}

Doing a bit of digging into that placeholder.invalid uri, I found the deprecated and destined(?) to be removed https://placeholder.invalid/wiki/Special:Badtitle, with a mention of T221368: cdnPurge and other jobs fail completely to execute

More digging, found this change to NotificationDeleteJob.php, which coincides with the Nov 2 increase...

On advisement of @RhinosF1, I agree this may warrant holding the upcoming train..

TheresNoTime added a parent task: T320515: 1.40.0-wmf.10 deployment blockers.Nov 5 2022, 8:33 AM

In T322342#8372147, @TheresNoTime wrote:

More digging, found this change to NotificationDeleteJob.php, which coincides with the Nov 2 increase...

@Ladsgroup, @Reedy: can you take a look?

RhinosF1 added a subscriber: Reedy.Nov 5 2022, 8:37 AM

Change 853470 had a related patch set uploaded (by Zabe; author: Zabe):

[mediawiki/extensions/Echo@master] Fix NotificationDeleteJob class name

https://gerrit.wikimedia.org/r/853470

gerritbot added a project: Patch-For-Review.Nov 5 2022, 9:39 AM

Zabe added a subscriber: Zabe.Nov 5 2022, 9:39 AM

Change 853470 merged by jenkins-bot:

[mediawiki/extensions/Echo@master] Fix NotificationDeleteJob class name

https://gerrit.wikimedia.org/r/853470

Maintenance_bot removed a project: Patch-For-Review.Nov 5 2022, 10:30 AM

ReleaseTaggerBot added a project: MW-1.40-notes (1.40.0-wmf.10; 2022-11-14).Nov 5 2022, 11:00 AM

Can this be resolved?

• Ladsgroup closed this task as Resolved.Nov 5 2022, 11:50 AM

• Ladsgroup assigned this task to Zabe.

Still getting Y2ZOP1rG9juoZpSmy0kEugAAAAQ] /w/index.php?returnto=Main+Page&title=Special:CreateAccount JobQueueError: Could not enqueue jobs when attempting to create an account on https://en.wikipedia.beta.wmflabs.org/wiki, so probably not..

Zabe reopened this task as Open.Nov 5 2022, 3:25 PM

Change 853474 had a related patch set uploaded (by Zabe; author: Zabe):

[mediawiki/extensions/Echo@master] Fix $wgJobClasses keys

https://gerrit.wikimedia.org/r/853474

gerritbot added a project: Patch-For-Review.Nov 5 2022, 3:29 PM

Change 853474 merged by jenkins-bot:

[mediawiki/extensions/Echo@master] Update job name to match $wgJobClasses key

https://gerrit.wikimedia.org/r/853474

Zabe closed this task as Resolved.Nov 5 2022, 4:59 PM

Maintenance_bot removed a project: Patch-For-Review.Nov 5 2022, 5:30 PM

There is still a constant trickle of Could not enqueue jobs for stream mediawiki.job.cirrusSearchCheckerJob errors.

RhinosF1 reopened this task as Open.Nov 6 2022, 4:28 PM