Page MenuHomePhabricator

InvalidArgumentException: Expected IJobSpecification objects
Open, HighPublicPRODUCTION ERROR

Description

Error
normalized_message
[{reqId}] {exception_url}   InvalidArgumentException: Expected IJobSpecification objects
exception.trace
from /srv/mediawiki/php-1.37.0-wmf.15/includes/jobqueue/JobQueueGroup.php(481)
#0 /srv/mediawiki/php-1.37.0-wmf.15/includes/jobqueue/JobQueueGroup.php(174): JobQueueGroup->assertValidJobs(array)
#1 /srv/mediawiki/php-1.37.0-wmf.15/includes/deferred/JobQueueEnqueueUpdate.php(58): JobQueueGroup->push(array)
#2 /srv/mediawiki/php-1.37.0-wmf.15/includes/deferred/DeferredUpdates.php(515): JobQueueEnqueueUpdate->doUpdate()
#3 /srv/mediawiki/php-1.37.0-wmf.15/includes/deferred/DeferredUpdates.php(391): DeferredUpdates::attemptUpdate(JobQueueEnqueueUpdate, Wikimedia\Rdbms\LBFactoryMulti)
#4 /srv/mediawiki/php-1.37.0-wmf.15/includes/deferred/DeferredUpdates.php(221): DeferredUpdates::run(JobQueueEnqueueUpdate, Wikimedia\Rdbms\LBFactoryMulti, Monolog\Logger, BufferingStatsdDataFactory, string)
#5 /srv/mediawiki/php-1.37.0-wmf.15/includes/deferred/DeferredUpdatesScope.php(267): DeferredUpdates::{closure}(JobQueueEnqueueUpdate, integer)
#6 /srv/mediawiki/php-1.37.0-wmf.15/includes/deferred/DeferredUpdatesScope.php(196): DeferredUpdatesScope->processStageQueue(integer, integer, Closure)
#7 /srv/mediawiki/php-1.37.0-wmf.15/includes/deferred/DeferredUpdates.php(242): DeferredUpdatesScope->processUpdates(integer, Closure)
#8 /srv/mediawiki/php-1.37.0-wmf.15/includes/MediaWiki.php(1136): DeferredUpdates::doUpdates(string)
#9 /srv/mediawiki/rpc/RunSingleJob.php(93): MediaWiki->restInPeace()
#10 {main}
Impact
  • Seeing a couple of these in production over the last four hours, unsure of user impact
Notes
  • Tracing the entire request via reqId in logstash determined that this was coming from shellbox

Details

Request URL
https://jobrunner.discovery.wmnet/rpc/RunSingleJob.php

Event Timeline

Seeing a couple of these in production over the last four hours, unsure of user impact

Probably some jobs are being dropped which is not great but ocassionally happens.

Tracing the entire request via reqId in logstash determined that this was coming from shellbox

I...don't see how Shellbox would do that, it receives requests, doesn't send them. To me it looks like the request is shelling out (to pygmentize, specifically), but in that same request something is enqueing an invalid job.

Tracing the entire request via reqId in logstash determined that this was coming from shellbox

I...don't see how Shellbox would do that, it receives requests, doesn't send them. To me it looks like the request is shelling out (to pygmentize, specifically), but in that same request something is enqueing an invalid job.

"coming from" is probably not the right way to say that. The reqID is passing through shellbox :)

If I tagged this one wrong, feel free to retag and apologies for the noise.

I'm not sure MediaWiki-Core-JobQueue is the right tag for this either, but I do think JobQueueGroup could provide a better error message to track down whatever is submitting invalid jobs. Also shouldn't being an instance of IJobSpecification checked when the job is queued rather than in post-send?

A instance of JobQueueEnqueueUpdate is created in JobQueueGroup::lazyPush after assertValidJobs call. The assertValidJobs call is also done for the real push inside JobQueueEnqueueUpdate, now failing.

That means the object must be corrupted some where between that. The class is a mergeable update, but it allows only itself to be merged. It looks impossible to get invalid things.

Change 713935 had a related patch set uploaded (by Umherirrender; author: Umherirrender):

[mediawiki/core@master] Improve exception message for JobQueueGroup::assertValidJobs

https://gerrit.wikimedia.org/r/713935

Change 713935 merged by jenkins-bot:

[mediawiki/core@master] Improve exception message for JobQueueGroup::assertValidJobs

https://gerrit.wikimedia.org/r/713935

The exception message was changed, please have a look at the logs if there are more informationen now.