OpenSearch indicates the response field is expected to be an object, but is provided something else.
"error"=>{"type"=>"mapper_parsing_exception", "reason"=>"object mapping for [response] tried to parse field [null] as object, but found a concrete value"}}}Likely due to ES issues raising the 503, response is sent as an array:
{
"type": "mediawiki",
"exception": {
"file": "/srv/mediawiki/php-1.41.0-wmf.24/extensions/EventBus/includes/Adapters/JobQueue/JobQueueEventBus.php:129",
"trace": "
from /srv/mediawiki/php-1.41.0-wmf.24/extensions/EventBus/includes/Adapters/JobQueue/JobQueueEventBus.php(129)
#0 /srv/mediawiki/php-1.41.0-wmf.24/includes/jobqueue/JobQueue.php(373): MediaWiki\Extension\EventBus\Adapters\JobQueue\JobQueueEventBus->doBatchPush(array, integer)
#1 /srv/mediawiki/php-1.41.0-wmf.24/includes/jobqueue/JobQueue.php(345): JobQueue->batchPush(array, integer)
#2 /srv/mediawiki/php-1.41.0-wmf.24/includes/jobqueue/JobQueueGroup.php(153): JobQueue->push(array)
#3 /srv/mediawiki/php-1.41.0-wmf.24/includes/deferred/JobQueueEnqueueUpdate.php(69): JobQueueGroup->push(array)
#4 /srv/mediawiki/php-1.41.0-wmf.24/includes/deferred/DeferredUpdates.php(498): JobQueueEnqueueUpdate->doUpdate()
#5 /srv/mediawiki/php-1.41.0-wmf.24/includes/deferred/DeferredUpdates.php(192): DeferredUpdates::attemptUpdate(JobQueueEnqueueUpdate, Wikimedia\Rdbms\LBFactoryMulti)
#6 /srv/mediawiki/php-1.41.0-wmf.24/includes/deferred/DeferredUpdates.php(298): DeferredUpdates::run(JobQueueEnqueueUpdate, string)
#7 /srv/mediawiki/php-1.41.0-wmf.24/includes/deferred/DeferredUpdatesScope.php(267): DeferredUpdates::{closure}(JobQueueEnqueueUpdate, integer)
#8 /srv/mediawiki/php-1.41.0-wmf.24/includes/deferred/DeferredUpdatesScope.php(196): DeferredUpdatesScope->processStageQueue(integer, integer, Closure)
#9 /srv/mediawiki/php-1.41.0-wmf.24/includes/deferred/DeferredUpdates.php(301): DeferredUpdatesScope->processUpdates(integer, Closure)
#10 /srv/mediawiki/php-1.41.0-wmf.24/includes/deferred/DeferredUpdatesScope.php(267): DeferredUpdates::{closure}(AutoCommitUpdate, integer)
#11 /srv/mediawiki/php-1.41.0-wmf.24/includes/deferred/DeferredUpdatesScope.php(196): DeferredUpdatesScope->processStageQueue(integer, integer, Closure)
#12 /srv/mediawiki/php-1.41.0-wmf.24/includes/deferred/DeferredUpdates.php(306): DeferredUpdatesScope->processUpdates(integer, Closure)
#13 /srv/mediawiki/php-1.41.0-wmf.24/extensions/EventBus/includes/JobExecutor.php(109): DeferredUpdates::doUpdates()
#14 /srv/mediawiki/rpc/RunSingleJob.php(77): MediaWiki\Extension\EventBus\JobExecutor->execute(array)
#15 {main}
",
"message": "Could not enqueue jobs",
"class": "JobQueueError",
"code": 0
},
"channel": "EventBus",
"program": "mediawiki",
"facility": "user",
"http_method": "POST",
"severity": "err",
"@version": 1,
"host": "mw1438",
"reqId": "2a420ff0-f34e-4aa3-935e-a32fc874c59a",
"logsource": "mw1438",
"referrer": null,
"normalized_message": "Could not enqueue jobs for stream {stream}",
"tags": [
"input-kafka-rsyslog-udp-localhost",
"rsyslog-udp-localhost",
"kafka",
"es",
"es",
"dlq"
],
"url": "/rpc/RunSingleJob.php",
"wiki": "enwiktionary",
"message": "Could not enqueue jobs for stream mediawiki.job.cirrusSearchLinksUpdate",
"monolog_level": 400,
"response": [
"Unable to deliver all events: 503: Service Unavailable"
],
"@timestamp": "2023-09-01T18:50:33.929Z",
"stream": "mediawiki.job.cirrusSearchLinksUpdate",
"timestamp": "2023-09-01T18:50:33+00:00",
"server": "jobrunner.discovery.wmnet",
"phpversion": "7.4.33",
"level": "ERROR",
"servergroup": "jobrunner",
"shard": "s2",
"mwversion": "1.41.0-wmf.24"
}Grafana graph of the event. This event negatively affects Logstash SLOs. ECS logs of the same message were ingested properly.