InvalidArgumentException: Expected IJobSpecification objects
Closed, DeclinedPublicPRODUCTION ERROR
Actions

Assigned To

None

Authored By

	thcipriani
	Jul 28 2021, 6:33 PM

Description

Error

mwversion: 1.37.0-wmf.15
reqId: 524f1dbf8bbfce16f81fee2b
Find reqId in Logstash
Find normalized_message in Logstash

normalized_message

[{reqId}] {exception_url}   InvalidArgumentException: Expected IJobSpecification objects

exception.trace

from /srv/mediawiki/php-1.37.0-wmf.15/includes/jobqueue/JobQueueGroup.php(481)
#0 /srv/mediawiki/php-1.37.0-wmf.15/includes/jobqueue/JobQueueGroup.php(174): JobQueueGroup->assertValidJobs(array)
#1 /srv/mediawiki/php-1.37.0-wmf.15/includes/deferred/JobQueueEnqueueUpdate.php(58): JobQueueGroup->push(array)
#2 /srv/mediawiki/php-1.37.0-wmf.15/includes/deferred/DeferredUpdates.php(515): JobQueueEnqueueUpdate->doUpdate()
#3 /srv/mediawiki/php-1.37.0-wmf.15/includes/deferred/DeferredUpdates.php(391): DeferredUpdates::attemptUpdate(JobQueueEnqueueUpdate, Wikimedia\Rdbms\LBFactoryMulti)
#4 /srv/mediawiki/php-1.37.0-wmf.15/includes/deferred/DeferredUpdates.php(221): DeferredUpdates::run(JobQueueEnqueueUpdate, Wikimedia\Rdbms\LBFactoryMulti, Monolog\Logger, BufferingStatsdDataFactory, string)
#5 /srv/mediawiki/php-1.37.0-wmf.15/includes/deferred/DeferredUpdatesScope.php(267): DeferredUpdates::{closure}(JobQueueEnqueueUpdate, integer)
#6 /srv/mediawiki/php-1.37.0-wmf.15/includes/deferred/DeferredUpdatesScope.php(196): DeferredUpdatesScope->processStageQueue(integer, integer, Closure)
#7 /srv/mediawiki/php-1.37.0-wmf.15/includes/deferred/DeferredUpdates.php(242): DeferredUpdatesScope->processUpdates(integer, Closure)
#8 /srv/mediawiki/php-1.37.0-wmf.15/includes/MediaWiki.php(1136): DeferredUpdates::doUpdates(string)
#9 /srv/mediawiki/rpc/RunSingleJob.php(93): MediaWiki->restInPeace()
#10 {main}

Impact

Seeing a couple of these in production over the last four hours, unsure of user impact

Notes

Tracing the entire request via reqId in logstash determined that this was coming from shellbox

Details

Request URL: https://jobrunner.discovery.wmnet/rpc/RunSingleJob.php

	Project	Branch	Lines +/-	Subject
	mediawiki/core	master	+2 -1	Improve exception message for JobQueueGroup::assertValidJobs

Customize query in gerrit

Related Objects

Mentioned Here: T245183: PHP7 corruption reports in 2020-2022 (Call on wrong object, etc.)

Event Timeline

thcipriani created this task.Jul 28 2021, 6:33 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 28 2021, 6:33 PM

thcipriani added projects: Platform Engineering, serviceops.Jul 28 2021, 6:44 PM

thcipriani moved this task from Untriaged to July 2021 on the Wikimedia-production-error board.

Seeing a couple of these in production over the last four hours, unsure of user impact

Probably some jobs are being dropped which is not great but ocassionally happens.

Tracing the entire request via reqId in logstash determined that this was coming from shellbox

I...don't see how Shellbox would do that, it receives requests, doesn't send them. To me it looks like the request is shelling out (to pygmentize, specifically), but in that same request something is enqueing an invalid job.

In T287623#7244272, @Legoktm wrote:

Tracing the entire request via reqId in logstash determined that this was coming from shellbox

I...don't see how Shellbox would do that, it receives requests, doesn't send them. To me it looks like the request is shelling out (to pygmentize, specifically), but in that same request something is enqueing an invalid job.

"coming from" is probably not the right way to say that. The reqID is passing through shellbox :)

If I tagged this one wrong, feel free to retag and apologies for the noise.

I'm not sure MediaWiki-Core-JobQueue is the right tag for this either, but I do think JobQueueGroup could provide a better error message to track down whatever is submitting invalid jobs. Also shouldn't being an instance of IJobSpecification checked when the job is queued rather than in post-send?

daniel triaged this task as High priority.Aug 19 2021, 4:23 PM

daniel edited projects, added Platform Team Workboards (Clinic Duty Team); removed Platform Engineering.

A instance of JobQueueEnqueueUpdate is created in JobQueueGroup::lazyPush after assertValidJobs call. The assertValidJobs call is also done for the real push inside JobQueueEnqueueUpdate, now failing.

That means the object must be corrupted some where between that. The class is a mergeable update, but it allows only itself to be merged. It looks impossible to get invalid things.

Change 713935 had a related patch set uploaded (by Umherirrender; author: Umherirrender):

[mediawiki/core@master] Improve exception message for JobQueueGroup::assertValidJobs

https://gerrit.wikimedia.org/r/713935

gerritbot added a project: Patch-For-Review.Aug 19 2021, 8:06 PM

Change 713935 merged by jenkins-bot:

[mediawiki/core@master] Improve exception message for JobQueueGroup::assertValidJobs

https://gerrit.wikimedia.org/r/713935

ReleaseTaggerBot added a project: MW-1.37-notes (1.37.0-wmf.20; 2021-08-23).Aug 19 2021, 10:00 PM

Maintenance_bot removed a project: Patch-For-Review.Aug 19 2021, 10:11 PM

The exception message was changed, please have a look at the logs if there are more informationen now.

Found one instance in the last month, reqId=e547ee30-2891-4c1f-a09b-5b48c310f11f from October 16. It says: InvalidArgumentException: Expected IJobSpecification objects, got unknown type.

"unknown type" is coming from PHP's builtin gettype according to https://gerrit.wikimedia.org/r/c/mediawiki/core/+/713935/. This indicates that something has gone wrong inside the guts of PHP...

Not sure if the garbage collector could destroy the object in this case or it is corruption like T245183?

Umherirrender removed a subscriber: Umherirrender.Nov 24 2021, 7:36 PM

Zero results for "Expected IJobSpecification" in the last 90 days on type:mediawiki in Logstash at WMF. Closing for now.

Krinkle edited projects, added MediaWiki-Platform-Team; removed Platform Team Workboards (Clinic Duty Team).Aug 1 2023, 4:06 PM

InvalidArgumentException: Expected IJobSpecification objectsClosed, DeclinedPublicPRODUCTION ERRORActions