Page MenuHomePhabricator

JobQueueErrors (Could not enqueue jobs) on Beta Cluster
Closed, ResolvedPublicBUG REPORT

Description

Steps to replicate the issue:

  • At a Beta Cluster wiki (tested at EnBeta and at WikifunctionsBeta) in any standard namespace
  • Visit a history page, or diff page
  • Click "Thank" for an edit.

What happens?:

  • Thank action failed (error code: internal_api_error_JobQueueError). Please try again.

What should have happened instead?:

  • Successfully send a Thank

image.png (252×458 px, 19 KB)


[Y2Qip-30XgbdpvgTbKuwBgAAAA0] Exception caught: Could not enqueue jobs

JobQueueError at /srv/mediawiki/php-master/extensions/EventBus/includes/Adapters/JobQueue/JobQueueEventBus.php(133) from /srv/mediawiki/php-master/extensions/EventBus/includes/Adapters/JobQueue/JobQueueEventBus.php(133)
#0 /srv/mediawiki/php-master/includes/jobqueue/JobQueue.php(376): MediaWiki\Extension\EventBus\Adapters\JobQueue\JobQueueEventBus->doBatchPush(array, integer)
#1 /srv/mediawiki/php-master/includes/jobqueue/JobQueue.php(348): JobQueue->batchPush(array, integer)
#2 /srv/mediawiki/php-master/includes/jobqueue/JobQueueGroup.php(166): JobQueue->push(array)
#3 /srv/mediawiki/php-master/extensions/Echo/includes/Controller/NotificationController.php(223): JobQueueGroup->push(array)
#4 /srv/mediawiki/php-master/extensions/Echo/includes/Controller/NotificationController.php(172): MediaWiki\Extension\Notifications\Controller\NotificationController::enqueueDeleteJob(array, EchoEvent)
#5 /srv/mediawiki/php-master/extensions/Echo/includes/model/Event.php(193): MediaWiki\Extension\Notifications\Controller\NotificationController::notify(EchoEvent, boolean)
#6 /srv/mediawiki/php-master/extensions/Thanks/includes/ApiCoreThank.php(235): EchoEvent::create(array)
#7 /srv/mediawiki/php-master/extensions/Thanks/includes/ApiCoreThank.php(90): MediaWiki\Extension\Thanks\ApiCoreThank->sendThanks(User, string, integer, string, User, string, Title, boolean)
#8 /srv/mediawiki/php-master/includes/api/ApiMain.php(1902): MediaWiki\Extension\Thanks\ApiCoreThank->execute()
#9 /srv/mediawiki/php-master/includes/api/ApiMain.php(877): ApiMain->executeAction()
#10 /srv/mediawiki/php-master/includes/api/ApiMain.php(848): ApiMain->executeActionWithErrorHandling()
#11 /srv/mediawiki/php-master/api.php(90): ApiMain->execute()
#12 /srv/mediawiki/php-master/api.php(45): wfApiMain()
#13 /srv/mediawiki/w/api.php(3): require(string)#14 {main}

Noting that in T322442: Account creation broken on beta, this is (now?) preventing account creation on the Beta Cluster.

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald Transcript
TheresNoTime renamed this task from Error with Thanks on Beta Cluster - internal_api_error_JobQueueError to Exception caught: Could not enqueue jobs when attempting to 'thank' on Beta Cluster.Nov 3 2022, 8:23 PM
TheresNoTime updated the task description. (Show Details)
Tgr added a project: Event-Platform.
Tgr added a subscriber: Tgr.

Seems to happen for many different kinds of jobs, starting on Oct 22 and then vastly increasing in volume on Nov 2.

TheresNoTime renamed this task from Exception caught: Could not enqueue jobs when attempting to 'thank' on Beta Cluster to JobQueueErrors (Could not enqueue jobs) on Beta Cluster.Nov 5 2022, 7:21 AM
TheresNoTime updated the task description. (Show Details)
TheresNoTime added a subscriber: vaughnwalters.

On attempting to create an account, I logged this event for req Y2YP1VrG9juoZpSmy0nhNQAAAAM — perhaps of note is the invalid topic mentioned?

{
   "invalid":[
      
   ],
   "error":[
      {
         "status":"error",
         "event":{
            "$schema":"/mediawiki/job/1.0.0",
            "meta":{
               "uri":"https://placeholder.invalid/wiki/Special:Badtitle",
               "request_id":"Y2YP1VrG9juoZpSmy0nhNQAAAAM",
               "id":"42c33faa-52f1-4db9-992d-a991e1156cfc",
               "dt":"2022-11-05T07:25:11Z",
               "domain":"en.wikipedia.beta.wmflabs.org",
               "stream":"mediawiki.job.MediaWiki\\Extension\\Notifications\\Jobs\\NotificationDeleteJob"
            },
            "database":"enwiki",
            "type":"MediaWiki\\Extension\\Notifications\\Jobs\\NotificationDeleteJob",
            "params":{
               "userIds":{
                  "99620":99620
               },
               "namespace":0,
               "title":"Main_Page",
               "requestId":"Y2YP1VrG9juoZpSmy0nhNQAAAAM"
            },
            "mediawiki_signature":"93cdb3cef48c632835127bd925b5f4676bf7113b"
         },
         "context":{
            "message":"invalid topic"
         }
      }
   ]
}

Doing a bit of digging into that placeholder.invalid uri, I found the deprecated and destined(?) to be removed https://placeholder.invalid/wiki/Special:Badtitle, with a mention of T221368: cdnPurge and other jobs fail completely to execute

More digging, found this change to NotificationDeleteJob.php, which coincides with the Nov 2 increase...

TheresNoTime triaged this task as Unbreak Now! priority.Nov 5 2022, 8:33 AM
TheresNoTime added a subscriber: RhinosF1.

On advisement of @RhinosF1, I agree this may warrant holding the upcoming train..

More digging, found this change to NotificationDeleteJob.php, which coincides with the Nov 2 increase...

@Ladsgroup, @Reedy: can you take a look?

Change 853470 had a related patch set uploaded (by Zabe; author: Zabe):

[mediawiki/extensions/Echo@master] Fix NotificationDeleteJob class name

https://gerrit.wikimedia.org/r/853470

Change 853470 merged by jenkins-bot:

[mediawiki/extensions/Echo@master] Fix NotificationDeleteJob class name

https://gerrit.wikimedia.org/r/853470

Ladsgroup assigned this task to Zabe.

Still getting Y2ZOP1rG9juoZpSmy0kEugAAAAQ] /w/index.php?returnto=Main+Page&title=Special:CreateAccount JobQueueError: Could not enqueue jobs when attempting to create an account on https://en.wikipedia.beta.wmflabs.org/wiki, so probably not..

Change 853474 had a related patch set uploaded (by Zabe; author: Zabe):

[mediawiki/extensions/Echo@master] Fix $wgJobClasses keys

https://gerrit.wikimedia.org/r/853474

Change 853474 merged by jenkins-bot:

[mediawiki/extensions/Echo@master] Update job name to match $wgJobClasses key

https://gerrit.wikimedia.org/r/853474

There is still a constant trickle of Could not enqueue jobs for stream mediawiki.job.cirrusSearchCheckerJob errors.

Those are unrelated to this task, created T322491 for them. (They are quite low rate and also older, so that is also not a train blocker.)