Page MenuHomePhabricator

PHP Fatal Error: Class undefined: JobExecutor (jobrunners try to run labswiki jobs)
Closed, ResolvedPublic



Request ID: 68e83d397bb1068efff4a69f

PHP Fatal Error from line 65 of /srv/mediawiki/rpc/RunSingleJob.php: Class undefined: JobExecutor

server: mw1296
wiki: labswiki
shard: wikitech
#0 /srv/mediawiki/rpc/RunSingleJob.php(65): NO_FUNCTION_GIVEN()
#1 {main}


Unknown. Looks like maybe jobs are queued the wrong way. If that's the case, it might be that certain updates intended for Wikitech are not being applied (with no obvious means for recovery).


First seen 2018-10-18 but only a couple times each day. Unclear what the source of the jobs are. The job parameters are not available in the logs because the process is failing before the log context is established.

Event Timeline

Krinkle created this task.Nov 7 2018, 12:52 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 7 2018, 12:52 AM

Change 474885 had a related patch set uploaded (by Mobrovac; owner: Mobrovac):
[operations/mediawiki-config@master] RunSingleJob: Check that JobExecutor has been loaded

mobrovac added a subscriber: mobrovac.

The above patch does not solve the problem, but it improves the situation in that it logs the offending event when this happens, which allow us to begin investigating.

Actually, I believe there's an easier way to find out the events that cause it:

The search shows that the job in question is BounceHandlerJob. Will continue the investigation.

Ok, I understand what's happening here. The BounceHandler extension sends the job cross-wiki, in this particular case to the wikitech wiki queue. However, JobQueueGroup::singleton()->get() uses global $wgJobTypeConf, which is different for all wikis except wikitech. That's how the event ends up in kafka and obviously fails cause wikitech doesn't support kafka job queue.

Thank you for the investigation @Pchelolo ! So once again we are hitting this global job conf problem.

Change 474885 merged by jenkins-bot:
[operations/mediawiki-config@master] RunSingleJob: Check that JobExecutor has been loaded

Mentioned in SAL (#wikimedia-operations) [2018-11-27T12:19:54Z] <mobrovac@deploy1001> Synchronized rpc/RunSingleJob.php: RunSingleJob: Check that JobExecutor has been loaded - T208922 (duration: 00m 47s)

Krinkle moved this task from Limbo to Watching on the Performance-Team (Radar) board.

Thanks for looking into this. Moving to our radar to keep tabs on. Do let me know if there's anything I can help with :)

mobrovac closed this task as Resolved.Apr 29 2019, 7:53 PM
mobrovac claimed this task.
mobrovac triaged this task as High priority.

This particular error has not appeared since, so closing this particular task.

mmodell changed the subtype of this task from "Task" to "Production Error".Aug 28 2019, 11:08 PM