Seeing a couple of these in production over the last four hours, unsure of user impact

Probably some jobs are being dropped which is not great but ocassionally happens.

Tracing the entire request via reqId in logstash determined that this was coming from shellbox

I...don't see how Shellbox would do that, it receives requests, doesn't send them. To me it looks like the request is shelling out (to pygmentize, specifically), but in that same request something is enqueing an invalid job.

In T287623#7244272, @Legoktm wrote:

Tracing the entire request via reqId in logstash determined that this was coming from shellbox

I...don't see how Shellbox would do that, it receives requests, doesn't send them. To me it looks like the request is shelling out (to pygmentize, specifically), but in that same request something is enqueing an invalid job.

"coming from" is probably not the right way to say that. The reqID is passing through shellbox :)

If I tagged this one wrong, feel free to retag and apologies for the noise.

I'm not sure MediaWiki-Core-JobQueue is the right tag for this either, but I do think JobQueueGroup could provide a better error message to track down whatever is submitting invalid jobs. Also shouldn't being an instance of IJobSpecification checked when the job is queued rather than in post-send?

A instance of JobQueueEnqueueUpdate is created in JobQueueGroup::lazyPush after assertValidJobs call. The assertValidJobs call is also done for the real push inside JobQueueEnqueueUpdate, now failing.

That means the object must be corrupted some where between that. The class is a mergeable update, but it allows only itself to be merged. It looks impossible to get invalid things.

Change 713935 had a related patch set uploaded (by Umherirrender; author: Umherirrender):

[mediawiki/core@master] Improve exception message for JobQueueGroup::assertValidJobs

https://gerrit.wikimedia.org/r/713935

Change 713935 merged by jenkins-bot:

[mediawiki/core@master] Improve exception message for JobQueueGroup::assertValidJobs

https://gerrit.wikimedia.org/r/713935

The exception message was changed, please have a look at the logs if there are more informationen now.

Found one instance in the last month, reqId=e547ee30-2891-4c1f-a09b-5b48c310f11f from October 16. It says: InvalidArgumentException: Expected IJobSpecification objects, got unknown type.

"unknown type" is coming from PHP's builtin gettype according to https://gerrit.wikimedia.org/r/c/mediawiki/core/+/713935/. This indicates that something has gone wrong inside the guts of PHP...

Not sure if the garbage collector could destroy the object in this case or it is corruption like T245183?

I have removed a log filter from Kibana MediaWiki New Errors dashboard which pointed back to this task. There is no hit over the last eight weeks.

The filter was T287623 - Expected IJobSpecification objects:

{
  "query": {
    "match_phrase": {
      "error.message": "Expected IJobSpecification objects"
    }
  }
}