Page MenuHomePhabricator

cirrusSearchCheckerJob JobQueueErrors (Could not enqueue jobs) on Beta Cluster
Open, Needs TriagePublicPRODUCTION ERROR

Description

[3af68f13367d1f4209b454e8] Could not enqueue jobs for stream mediawiki.job.cirrusSearchCheckerJob

from /srv/mediawiki/php-master/extensions/EventBus/includes/Adapters/JobQueue/JobQueueEventBus.php(129)
#0 /srv/mediawiki/php-master/includes/jobqueue/JobQueue.php(376): MediaWiki\Extension\EventBus\Adapters\JobQueue\JobQueueEventBus->doBatchPush(array, integer)
#1 /srv/mediawiki/php-master/includes/jobqueue/JobQueue.php(348): JobQueue->batchPush(array, integer)
#2 /srv/mediawiki/php-master/includes/jobqueue/JobQueueGroup.php(166): JobQueue->push(array)
#3 /srv/mediawiki/php-master/extensions/CirrusSearch/maintenance/SaneitizeJobs.php(250): JobQueueGroup->push(array)
#4 /srv/mediawiki/php-master/extensions/CirrusSearch/maintenance/SaneitizeJobs.php(89): CirrusSearch\Maintenance\SaneitizeJobs->pushJobs()
#5 /srv/mediawiki/php-master/maintenance/includes/MaintenanceRunner.php(309): CirrusSearch\Maintenance\SaneitizeJobs->execute()
#6 /srv/mediawiki/php-master/maintenance/doMaintenance.php(85): MediaWiki\Maintenance\MaintenanceRunner->run()
#7 /srv/mediawiki/php-master/extensions/CirrusSearch/maintenance/SaneitizeJobs.php(423): require_once(string)
#8 /srv/mediawiki/multiversion/MWScript.php(120): require_once(string)
#9 {main}

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Gehel subscribed.

Removing the Search team from this, it looks to be related to JobQueue and not touching Search yet. Please add us again if investigation shows otherwise.

Change 899883 had a related patch set uploaded (by Jforrester; author: Jforrester):

[mediawiki/extensions/EventBus@master] JobQueueEventBus: If sending the job fails, add the result for debugging

https://gerrit.wikimedia.org/r/899883

Change 899883 merged by jenkins-bot:

[mediawiki/extensions/EventBus@master] JobQueueEventBus: If sending the job fails, add the result for debugging

https://gerrit.wikimedia.org/r/899883

@Jdforrester-WMF @Legoktm @Ottomata Adding the result to the response key has led to mapper_parsing_exception on logstash: object mapping for [response] tried to parse field [null] as object, but found a concrete value

The error message is somewhat misleading because response is not null in these logs, but is an array when an object is what was expected.

{
  "wiki": "commonswiki",
  "phpversion": "7.4.33",
  "channel": "EventBus",
  "program": "mediawiki",
  "reqId": "feaf6597-3e4a-4b05-97fe-adec394ab1aa",
  "server": "commons.wikimedia.org",
  "servergroup": "kube-mw-api-int",
  "url": "/w/rest.php/v1/page/File%3ABalance_Farm_(Titley)_-_geograph.org.uk_-_5870582.jpg/with_html",
  "response": [
    "Unable to deliver all events: 503: Service Unavailable"
  ],
  "tags": [
    "input-kafka-rsyslog-udp-localhost",
    "rsyslog-udp-localhost",
    "kafka",
    "es",
    "es",
    "dlq"
  ],
  "level": "ERROR",
  "exception": {
    "code": 0,
    "class": "JobQueueError",
    "message": "Could not enqueue jobs",
    "file": "/srv/mediawiki/php-1.43.0-wmf.7/extensions/EventBus/includes/Adapters/JobQueue/JobQueueEventBus.php:129",
    "trace": "from /srv/mediawiki/php-1.43.0-wmf.7/extensions/EventBus/includes/Adapters/JobQueue/JobQueueEventBus.php(129)\n#0 /srv/mediawiki/php-1.43.0-wmf.7/includes/jobqueue/JobQueue.php(380): MediaWiki\\Extension\\EventBus\\Adapters\\JobQueue\\JobQueueEventBus->doBatchPush(array, integer)\n#1 /srv/mediawiki/php-1.43.0-wmf.7/includes/jobqueue/JobQueue.php(352): JobQueue->batchPush(array, integer)\n#2 /srv/mediawiki/php-1.43.0-wmf.7/includes/jobqueue/JobQueueGroup.php(155): JobQueue->push(array)\n#3 /srv/mediawiki/php-1.43.0-wmf.7/extensions/Linter/includes/Hooks.php(295): JobQueueGroup->push(array)\n#4 /srv/mediawiki/php-1.43.0-wmf.7/includes/HookContainer/HookContainer.php(159): MediaWiki\\Linter\\Hooks->onParserLogLinterData(string, integer, array)\n#5 /srv/mediawiki/php-1.43.0-wmf.7/includes/HookContainer/HookRunner.php(2997): MediaWiki\\HookContainer\\HookContainer->run(string, array)\n#6 /srv/mediawiki/php-1.43.0-wmf.7/includes/parser/Parsoid/Config/DataAccess.php(460): MediaWiki\\HookContainer\\HookRunner->onParserLogLinterData(string, integer, array)\n#7 /srv/mediawiki/php-1.43.0-wmf.7/vendor/wikimedia/parsoid/src/Logger/LintLogger.php(103): MediaWiki\\Parser\\Parsoid\\Config\\DataAccess->logLinterData(MediaWiki\\Parser\\Parsoid\\Config\\PageConfig, array)\n#8 /srv/mediawiki/php-1.43.0-wmf.7/vendor/wikimedia/parsoid/src/Parsoid.php(244): Wikimedia\\Parsoid\\Logger\\LintLogger->logLintOutput()\n#9 /srv/mediawiki/php-1.43.0-wmf.7/includes/parser/Parsoid/ParsoidParser.php(149): Wikimedia\\Parsoid\\Parsoid->wikitext2html(MediaWiki\\Parser\\Parsoid\\Config\\PageConfig, array, NULL, MediaWiki\\Parser\\ParserOutput)\n#10 /srv/mediawiki/php-1.43.0-wmf.7/includes/parser/Parsoid/ParsoidParser.php(256): MediaWiki\\Parser\\Parsoid\\ParsoidParser->genParserOutput(MediaWiki\\Parser\\Parsoid\\Config\\PageConfig, ParserOptions)\n#11 /srv/mediawiki/php-1.43.0-wmf.7/includes/content/WikitextContentHandler.php(377): MediaWiki\\Parser\\Parsoid\\ParsoidParser->parse(string, MediaWiki\\Title\\Title, ParserOptions, boolean, boolean, integer)\n#12 /srv/mediawiki/php-1.43.0-wmf.7/includes/content/ContentHandler.php(1675): WikitextContentHandler->fillParserOutput(WikitextContent, MediaWiki\\Content\\Renderer\\ContentParseParams, MediaWiki\\Parser\\ParserOutput)\n#13 /srv/mediawiki/php-1.43.0-wmf.7/includes/content/Renderer/ContentRenderer.php(67): ContentHandler->getParserOutput(WikitextContent, MediaWiki\\Content\\Renderer\\ContentParseParams)\n#14 /srv/mediawiki/php-1.43.0-wmf.7/includes/Revision/RenderedRevision.php(260): MediaWiki\\Content\\Renderer\\ContentRenderer->getParserOutput(WikitextContent, MediaWiki\\Page\\PageStoreRecord, MediaWiki\\Revision\\RevisionStoreRecord, ParserOptions, boolean)\n#15 /srv/mediawiki/php-1.43.0-wmf.7/includes/Revision/RenderedRevision.php(232): MediaWiki\\Revision\\RenderedRevision->getSlotParserOutputUncached(WikitextContent, boolean)\n#16 /srv/mediawiki/php-1.43.0-wmf.7/includes/Revision/RevisionRenderer.php(226): MediaWiki\\Revision\\RenderedRevision->getSlotParserOutput(string, array)\n#17 /srv/mediawiki/php-1.43.0-wmf.7/includes/Revision/RevisionRenderer.php(164): MediaWiki\\Revision\\RevisionRenderer->combineSlotOutput(MediaWiki\\Revision\\RenderedRevision, ParserOptions, array)\n#18 [internal function]: MediaWiki\\Revision\\RevisionRenderer->MediaWiki\\Revision\\{closure}(MediaWiki\\Revision\\RenderedRevision, array)\n#19 /srv/mediawiki/php-1.43.0-wmf.7/includes/Revision/RenderedRevision.php(199): call_user_func(Closure, MediaWiki\\Revision\\RenderedRevision, array)\n#20 /srv/mediawiki/php-1.43.0-wmf.7/includes/page/ParserOutputAccess.php(406): MediaWiki\\Revision\\RenderedRevision->getRevisionParserOutput()\n#21 /srv/mediawiki/php-1.43.0-wmf.7/includes/page/ParserOutputAccess.php(357): MediaWiki\\Page\\ParserOutputAccess->renderRevision(MediaWiki\\Page\\PageStoreRecord, ParserOptions, MediaWiki\\Revision\\RevisionStoreRecord, integer)\n#22 /srv/mediawiki/php-1.43.0-wmf.7/includes/parser/Parsoid/ParsoidOutputAccess.php(108): MediaWiki\\Page\\ParserOutputAccess->getParserOutput(MediaWiki\\Page\\PageStoreRecord, ParserOptions, MediaWiki\\Revision\\RevisionStoreRecord, integer)\n#23 /srv/mediawiki/php-1.43.0-wmf.7/includes/Rest/Handler/Helper/HtmlOutputRendererHelper.php(787): MediaWiki\\Parser\\Parsoid\\ParsoidOutputAccess->getParserOutput(MediaWiki\\Page\\PageStoreRecord, ParserOptions, MediaWiki\\Revision\\RevisionStoreRecord, integer, boolean)\n#24 /srv/mediawiki/php-1.43.0-wmf.7/includes/Rest/Handler/Helper/HtmlOutputRendererHelper.php(591): MediaWiki\\Rest\\Handler\\Helper\\HtmlOutputRendererHelper->getParserOutputInternal(ParserOptions)\n#25 /srv/mediawiki/php-1.43.0-wmf.7/includes/Rest/Handler/Helper/HtmlOutputRendererHelper.php(516): MediaWiki\\Rest\\Handler\\Helper\\HtmlOutputRendererHelper->getParserOutput()\n#26 /srv/mediawiki/php-1.43.0-wmf.7/includes/Rest/Handler/PageHTMLHandler.php(154): MediaWiki\\Rest\\Handler\\Helper\\HtmlOutputRendererHelper->getETag(string)\n#27 /srv/mediawiki/php-1.43.0-wmf.7/includes/Rest/Handler.php(456): MediaWiki\\Rest\\Handler\\PageHTMLHandler->getETag()\n#28 /srv/mediawiki/php-1.43.0-wmf.7/includes/Rest/Handler.php(473): MediaWiki\\Rest\\Handler->getConditionalHeaderUtil()\n#29 /srv/mediawiki/php-1.43.0-wmf.7/includes/Rest/Module/Module.php(358): MediaWiki\\Rest\\Handler->checkPreconditions()\n#30 /srv/mediawiki/php-1.43.0-wmf.7/includes/Rest/Module/Module.php(265): MediaWiki\\Rest\\Module\\Module->executeHandler(MediaWiki\\Rest\\Handler\\PageHTMLHandler)\n#31 /srv/mediawiki/php-1.43.0-wmf.7/includes/Rest/Router.php(493): MediaWiki\\Rest\\Module\\Module->execute(string, MediaWiki\\Rest\\RequestFromGlobals)\n#32 /srv/mediawiki/php-1.43.0-wmf.7/includes/Rest/Router.php(470): MediaWiki\\Rest\\Router->doExecute(string, MediaWiki\\Rest\\RequestFromGlobals)\n#33 /srv/mediawiki/php-1.43.0-wmf.7/includes/Rest/EntryPoint.php(218): MediaWiki\\Rest\\Router->execute(MediaWiki\\Rest\\RequestFromGlobals)\n#34 /srv/mediawiki/php-1.43.0-wmf.7/includes/MediaWikiEntryPoint.php(200): MediaWiki\\Rest\\EntryPoint->execute()\n#35 /srv/mediawiki/php-1.43.0-wmf.7/rest.php(38): MediaWiki\\MediaWikiEntryPoint->run()\n#36 /srv/mediawiki/w/rest.php(3): require(string)\n#37 {main}"
  },
  "message": "Could not enqueue jobs for stream mediawiki.job.RecordLintJob",
  "referrer": null,
  "logsource": "mw-api-int.eqiad.main-dcd594996-988lp",
  "host": "mw-api-int.eqiad.main-dcd594996-988lp",
  "http_method": "GET",
  "@timestamp": "2024-06-04T15:05:29.888Z",
  "type": "mediawiki",
  "normalized_message": "Could not enqueue jobs for stream {stream}",
  "facility": "user",
  "kubernetes": {
    "pod_name": "mw-api-int.eqiad.main-dcd594996-988lp",
    "namespace_name": "mw-api-int",
    "host": "mw1422.eqiad.wmnet",
    "labels": {
      "deployment": "mw-api-int",
      "release": "main"
    }
  },
  "shard": "s4",
  "mwversion": "1.43.0-wmf.7",
  "timestamp": "2024-06-04T15:05:29+00:00",
  "severity": "err",
  "stream": "mediawiki.job.RecordLintJob",
  "monolog_level": 400,
  "@version": 1
}

I suggest we either transform $result into an object or choose another field.

@Jdforrester-WMF @Legoktm @Ottomata Adding the result to the response key has led to mapper_parsing_exception on logstash: object mapping for [response] tried to parse field [null] as object, but found a concrete value

The error message is somewhat misleading because response is not null in these logs, but is an array when an object is what was expected.

Ah, is 'response' a reserved key? We can inject it as 'invalidResponse' or whatever instead?

[…]

I suggest we either transform $result into an object or choose another field.

Ah, is 'response' a reserved key?

response is not a reserved key. All emitters of the response key in logs need to agree on the data type because OpenSearch is not tolerant of data type mismatches. Link in case you're interested in further reading about the remedy.

We can inject it as 'invalidResponse' or whatever instead?

That should work. As long as response is an object, the events will stop being dead-lettered.

Change #1038873 had a related patch set uploaded (by Jforrester; author: Jforrester):

[mediawiki/extensions/EventBus@master] JobQueueEventBus: Log invalid result as its own key, pushed as an object

https://gerrit.wikimedia.org/r/1038873

Change #1038873 merged by jenkins-bot:

[mediawiki/extensions/EventBus@master] JobQueueEventBus: Log invalid result as its own key, pushed as an object

https://gerrit.wikimedia.org/r/1038873

Change #1096667 had a related patch set uploaded (by Paladox; author: Jforrester):

[mediawiki/extensions/EventBus@REL1_42] JobQueueEventBus: Log invalid result as its own key, pushed as an object

https://gerrit.wikimedia.org/r/1096667

Change #1096667 merged by jenkins-bot:

[mediawiki/extensions/EventBus@REL1_42] JobQueueEventBus: Log invalid result as its own key, pushed as an object

https://gerrit.wikimedia.org/r/1096667

A_smart_kitten subscribed.

Adding Wikimedia-production-error, as per that tag’s description it can also be used to track Beta Cluster errors

Restricted Application changed the subtype of this task from "Task" to "Production Error". · View Herald TranscriptMar 21 2025, 5:39 AM