cirrusSearchCheckerJob JobQueueErrors (Could not enqueue jobs) on Beta Cluster
Open, Needs TriagePublicPRODUCTION ERROR
Actions

Description

[3af68f13367d1f4209b454e8] Could not enqueue jobs for stream mediawiki.job.cirrusSearchCheckerJob

from /srv/mediawiki/php-master/extensions/EventBus/includes/Adapters/JobQueue/JobQueueEventBus.php(129)
#0 /srv/mediawiki/php-master/includes/jobqueue/JobQueue.php(376): MediaWiki\Extension\EventBus\Adapters\JobQueue\JobQueueEventBus->doBatchPush(array, integer)
#1 /srv/mediawiki/php-master/includes/jobqueue/JobQueue.php(348): JobQueue->batchPush(array, integer)
#2 /srv/mediawiki/php-master/includes/jobqueue/JobQueueGroup.php(166): JobQueue->push(array)
#3 /srv/mediawiki/php-master/extensions/CirrusSearch/maintenance/SaneitizeJobs.php(250): JobQueueGroup->push(array)
#4 /srv/mediawiki/php-master/extensions/CirrusSearch/maintenance/SaneitizeJobs.php(89): CirrusSearch\Maintenance\SaneitizeJobs->pushJobs()
#5 /srv/mediawiki/php-master/maintenance/includes/MaintenanceRunner.php(309): CirrusSearch\Maintenance\SaneitizeJobs->execute()
#6 /srv/mediawiki/php-master/maintenance/doMaintenance.php(85): MediaWiki\Maintenance\MaintenanceRunner->run()
#7 /srv/mediawiki/php-master/extensions/CirrusSearch/maintenance/SaneitizeJobs.php(423): require_once(string)
#8 /srv/mediawiki/multiversion/MWScript.php(120): require_once(string)
#9 {main}

Details

Related Changes in Gerrit:

Subject	Repo	Branch	Lines +/-
JobQueueEventBus: Log invalid result as its own key, pushed as an object	mediawiki/extensions/EventBus	REL1_42	+1 -1
JobQueueEventBus: Log invalid result as its own key, pushed as an object	mediawiki/extensions/EventBus	master	+1 -1
JobQueueEventBus: If sending the job fails, add the result for debugging	mediawiki/extensions/EventBus	master	+1 -0

Customize query in gerrit

Related Objects

Mentioned In: T322342: JobQueueErrors (Could not enqueue jobs) on Beta Cluster
Mentioned Here: T363587: [Event Platform] Instrument EventBus with prometheus MW Statslib

Event Timeline

Zabe created this task.Nov 6 2022, 7:32 PM

Restricted Application added projects: Data-Engineering, Discovery-Search. · View Herald TranscriptNov 6 2022, 7:32 PM

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

Zabe mentioned this in T322342: JobQueueErrors (Could not enqueue jobs) on Beta Cluster.Nov 6 2022, 7:32 PM

• EChetty edited projects, added Data-Engineering-Planning; removed Data-Engineering.Nov 7 2022, 10:27 AM

• EChetty moved this task from Backlog to Ops Week on the Data-Engineering-Planning board.Nov 7 2022, 10:29 AM

• EChetty moved this task from Ops Week to Event Platform on the Data-Engineering-Planning board.Nov 7 2022, 11:13 AM

Removing the Search team from this, it looks to be related to JobQueue and not touching Search yet. Please add us again if investigation shows otherwise.

Change 899883 had a related patch set uploaded (by Jforrester; author: Jforrester):

[mediawiki/extensions/EventBus@master] JobQueueEventBus: If sending the job fails, add the result for debugging

https://gerrit.wikimedia.org/r/899883

gerritbot added a project: Patch-For-Review.Mar 16 2023, 12:13 AM

Change 899883 merged by jenkins-bot:

[mediawiki/extensions/EventBus@master] JobQueueEventBus: If sending the job fails, add the result for debugging

https://gerrit.wikimedia.org/r/899883

ReleaseTaggerBot added a project: MW-1.41-notes (1.41.0-wmf.12; 2023-06-06).May 31 2023, 8:00 AM

Maintenance_bot removed a project: Patch-For-Review.May 31 2023, 8:10 AM

JArguello-WMF removed a project: Data-Engineering-Planning.Jun 29 2023, 9:48 PM

Restricted Application added a project: Data-Engineering. · View Herald TranscriptJun 29 2023, 9:48 PM

JArguello-WMF moved this task from Incoming (new tickets) to Event Platform Backlog on the Data-Engineering board.Jun 29 2023, 10:22 PM

JArguello-WMF added a project: Data Engineering and Event Platform Team.Jun 30 2023, 4:29 PM

JArguello-WMF moved this task from Data Eng Backlog to Event Platform Backlog on the Data Engineering and Event Platform Team board.Jun 30 2023, 4:38 PM

• lbowmaker removed a project: Data Engineering and Event Platform Team.Nov 10 2023, 2:29 PM

@Jdforrester-WMF @Legoktm @Ottomata Adding the result to the response key has led to mapper_parsing_exception on logstash: object mapping for [response] tried to parse field [null] as object, but found a concrete value

The error message is somewhat misleading because response is not null in these logs, but is an array when an object is what was expected.

{
  "wiki": "commonswiki",
  "phpversion": "7.4.33",
  "channel": "EventBus",
  "program": "mediawiki",
  "reqId": "feaf6597-3e4a-4b05-97fe-adec394ab1aa",
  "server": "commons.wikimedia.org",
  "servergroup": "kube-mw-api-int",
  "url": "/w/rest.php/v1/page/File%3ABalance_Farm_(Titley)_-_geograph.org.uk_-_5870582.jpg/with_html",
  "response": [
    "Unable to deliver all events: 503: Service Unavailable"
  ],
  "tags": [
    "input-kafka-rsyslog-udp-localhost",
    "rsyslog-udp-localhost",
    "kafka",
    "es",
    "es",
    "dlq"
  ],
  "level": "ERROR",
  "exception": {
    "code": 0,
    "class": "JobQueueError",
    "message": "Could not enqueue jobs",
    "file": "/srv/mediawiki/php-1.43.0-wmf.7/extensions/EventBus/includes/Adapters/JobQueue/JobQueueEventBus.php:129",
    "trace": "from /srv/mediawiki/php-1.43.0-wmf.7/extensions/EventBus/includes/Adapters/JobQueue/JobQueueEventBus.php(129)\n#0 /srv/mediawiki/php-1.43.0-wmf.7/includes/jobqueue/JobQueue.php(380): MediaWiki\\Extension\\EventBus\\Adapters\\JobQueue\\JobQueueEventBus->doBatchPush(array, integer)\n#1 /srv/mediawiki/php-1.43.0-wmf.7/includes/jobqueue/JobQueue.php(352): JobQueue->batchPush(array, integer)\n#2 /srv/mediawiki/php-1.43.0-wmf.7/includes/jobqueue/JobQueueGroup.php(155): JobQueue->push(array)\n#3 /srv/mediawiki/php-1.43.0-wmf.7/extensions/Linter/includes/Hooks.php(295): JobQueueGroup->push(array)\n#4 /srv/mediawiki/php-1.43.0-wmf.7/includes/HookContainer/HookContainer.php(159): MediaWiki\\Linter\\Hooks->onParserLogLinterData(string, integer, array)\n#5 /srv/mediawiki/php-1.43.0-wmf.7/includes/HookContainer/HookRunner.php(2997): MediaWiki\\HookContainer\\HookContainer->run(string, array)\n#6 /srv/mediawiki/php-1.43.0-wmf.7/includes/parser/Parsoid/Config/DataAccess.php(460): MediaWiki\\HookContainer\\HookRunner->onParserLogLinterData(string, integer, array)\n#7 /srv/mediawiki/php-1.43.0-wmf.7/vendor/wikimedia/parsoid/src/Logger/LintLogger.php(103): MediaWiki\\Parser\\Parsoid\\Config\\DataAccess->logLinterData(MediaWiki\\Parser\\Parsoid\\Config\\PageConfig, array)\n#8 /srv/mediawiki/php-1.43.0-wmf.7/vendor/wikimedia/parsoid/src/Parsoid.php(244): Wikimedia\\Parsoid\\Logger\\LintLogger->logLintOutput()\n#9 /srv/mediawiki/php-1.43.0-wmf.7/includes/parser/Parsoid/ParsoidParser.php(149): Wikimedia\\Parsoid\\Parsoid->wikitext2html(MediaWiki\\Parser\\Parsoid\\Config\\PageConfig, array, NULL, MediaWiki\\Parser\\ParserOutput)\n#10 /srv/mediawiki/php-1.43.0-wmf.7/includes/parser/Parsoid/ParsoidParser.php(256): MediaWiki\\Parser\\Parsoid\\ParsoidParser->genParserOutput(MediaWiki\\Parser\\Parsoid\\Config\\PageConfig, ParserOptions)\n#11 /srv/mediawiki/php-1.43.0-wmf.7/includes/content/WikitextContentHandler.php(377): MediaWiki\\Parser\\Parsoid\\ParsoidParser->parse(string, MediaWiki\\Title\\Title, ParserOptions, boolean, boolean, integer)\n#12 /srv/mediawiki/php-1.43.0-wmf.7/includes/content/ContentHandler.php(1675): WikitextContentHandler->fillParserOutput(WikitextContent, MediaWiki\\Content\\Renderer\\ContentParseParams, MediaWiki\\Parser\\ParserOutput)\n#13 /srv/mediawiki/php-1.43.0-wmf.7/includes/content/Renderer/ContentRenderer.php(67): ContentHandler->getParserOutput(WikitextContent, MediaWiki\\Content\\Renderer\\ContentParseParams)\n#14 /srv/mediawiki/php-1.43.0-wmf.7/includes/Revision/RenderedRevision.php(260): MediaWiki\\Content\\Renderer\\ContentRenderer->getParserOutput(WikitextContent, MediaWiki\\Page\\PageStoreRecord, MediaWiki\\Revision\\RevisionStoreRecord, ParserOptions, boolean)\n#15 /srv/mediawiki/php-1.43.0-wmf.7/includes/Revision/RenderedRevision.php(232): MediaWiki\\Revision\\RenderedRevision->getSlotParserOutputUncached(WikitextContent, boolean)\n#16 /srv/mediawiki/php-1.43.0-wmf.7/includes/Revision/RevisionRenderer.php(226): MediaWiki\\Revision\\RenderedRevision->getSlotParserOutput(string, array)\n#17 /srv/mediawiki/php-1.43.0-wmf.7/includes/Revision/RevisionRenderer.php(164): MediaWiki\\Revision\\RevisionRenderer->combineSlotOutput(MediaWiki\\Revision\\RenderedRevision, ParserOptions, array)\n#18 [internal function]: MediaWiki\\Revision\\RevisionRenderer->MediaWiki\\Revision\\{closure}(MediaWiki\\Revision\\RenderedRevision, array)\n#19 /srv/mediawiki/php-1.43.0-wmf.7/includes/Revision/RenderedRevision.php(199): call_user_func(Closure, MediaWiki\\Revision\\RenderedRevision, array)\n#20 /srv/mediawiki/php-1.43.0-wmf.7/includes/page/ParserOutputAccess.php(406): MediaWiki\\Revision\\RenderedRevision->getRevisionParserOutput()\n#21 /srv/mediawiki/php-1.43.0-wmf.7/includes/page/ParserOutputAccess.php(357): MediaWiki\\Page\\ParserOutputAccess->renderRevision(MediaWiki\\Page\\PageStoreRecord, ParserOptions, MediaWiki\\Revision\\RevisionStoreRecord, integer)\n#22 /srv/mediawiki/php-1.43.0-wmf.7/includes/parser/Parsoid/ParsoidOutputAccess.php(108): MediaWiki\\Page\\ParserOutputAccess->getParserOutput(MediaWiki\\Page\\PageStoreRecord, ParserOptions, MediaWiki\\Revision\\RevisionStoreRecord, integer)\n#23 /srv/mediawiki/php-1.43.0-wmf.7/includes/Rest/Handler/Helper/HtmlOutputRendererHelper.php(787): MediaWiki\\Parser\\Parsoid\\ParsoidOutputAccess->getParserOutput(MediaWiki\\Page\\PageStoreRecord, ParserOptions, MediaWiki\\Revision\\RevisionStoreRecord, integer, boolean)\n#24 /srv/mediawiki/php-1.43.0-wmf.7/includes/Rest/Handler/Helper/HtmlOutputRendererHelper.php(591): MediaWiki\\Rest\\Handler\\Helper\\HtmlOutputRendererHelper->getParserOutputInternal(ParserOptions)\n#25 /srv/mediawiki/php-1.43.0-wmf.7/includes/Rest/Handler/Helper/HtmlOutputRendererHelper.php(516): MediaWiki\\Rest\\Handler\\Helper\\HtmlOutputRendererHelper->getParserOutput()\n#26 /srv/mediawiki/php-1.43.0-wmf.7/includes/Rest/Handler/PageHTMLHandler.php(154): MediaWiki\\Rest\\Handler\\Helper\\HtmlOutputRendererHelper->getETag(string)\n#27 /srv/mediawiki/php-1.43.0-wmf.7/includes/Rest/Handler.php(456): MediaWiki\\Rest\\Handler\\PageHTMLHandler->getETag()\n#28 /srv/mediawiki/php-1.43.0-wmf.7/includes/Rest/Handler.php(473): MediaWiki\\Rest\\Handler->getConditionalHeaderUtil()\n#29 /srv/mediawiki/php-1.43.0-wmf.7/includes/Rest/Module/Module.php(358): MediaWiki\\Rest\\Handler->checkPreconditions()\n#30 /srv/mediawiki/php-1.43.0-wmf.7/includes/Rest/Module/Module.php(265): MediaWiki\\Rest\\Module\\Module->executeHandler(MediaWiki\\Rest\\Handler\\PageHTMLHandler)\n#31 /srv/mediawiki/php-1.43.0-wmf.7/includes/Rest/Router.php(493): MediaWiki\\Rest\\Module\\Module->execute(string, MediaWiki\\Rest\\RequestFromGlobals)\n#32 /srv/mediawiki/php-1.43.0-wmf.7/includes/Rest/Router.php(470): MediaWiki\\Rest\\Router->doExecute(string, MediaWiki\\Rest\\RequestFromGlobals)\n#33 /srv/mediawiki/php-1.43.0-wmf.7/includes/Rest/EntryPoint.php(218): MediaWiki\\Rest\\Router->execute(MediaWiki\\Rest\\RequestFromGlobals)\n#34 /srv/mediawiki/php-1.43.0-wmf.7/includes/MediaWikiEntryPoint.php(200): MediaWiki\\Rest\\EntryPoint->execute()\n#35 /srv/mediawiki/php-1.43.0-wmf.7/rest.php(38): MediaWiki\\MediaWikiEntryPoint->run()\n#36 /srv/mediawiki/w/rest.php(3): require(string)\n#37 {main}"
  },
  "message": "Could not enqueue jobs for stream mediawiki.job.RecordLintJob",
  "referrer": null,
  "logsource": "mw-api-int.eqiad.main-dcd594996-988lp",
  "host": "mw-api-int.eqiad.main-dcd594996-988lp",
  "http_method": "GET",
  "@timestamp": "2024-06-04T15:05:29.888Z",
  "type": "mediawiki",
  "normalized_message": "Could not enqueue jobs for stream {stream}",
  "facility": "user",
  "kubernetes": {
    "pod_name": "mw-api-int.eqiad.main-dcd594996-988lp",
    "namespace_name": "mw-api-int",
    "host": "mw1422.eqiad.wmnet",
    "labels": {
      "deployment": "mw-api-int",
      "release": "main"
    }
  },
  "shard": "s4",
  "mwversion": "1.43.0-wmf.7",
  "timestamp": "2024-06-04T15:05:29+00:00",
  "severity": "err",
  "stream": "mediawiki.job.RecordLintJob",
  "monolog_level": 400,
  "@version": 1
}

I suggest we either transform $result into an object or choose another field.

colewhite added a subscriber: Ottomata.Jun 4 2024, 3:20 PM

In T322491#9860211, @colewhite wrote:

@Jdforrester-WMF @Legoktm @Ottomata Adding the result to the response key has led to mapper_parsing_exception on logstash: object mapping for [response] tried to parse field [null] as object, but found a concrete value

The error message is somewhat misleading because response is not null in these logs, but is an array when an object is what was expected.

Ah, is 'response' a reserved key? We can inject it as 'invalidResponse' or whatever instead?

[…]

I suggest we either transform $result into an object or choose another field.

In T322491#9860507, @Jdforrester-WMF wrote:

Ah, is 'response' a reserved key?

response is not a reserved key. All emitters of the response key in logs need to agree on the data type because OpenSearch is not tolerant of data type mismatches. Link in case you're interested in further reading about the remedy.

We can inject it as 'invalidResponse' or whatever instead?

That should work. As long as response is an object, the events will stop being dead-lettered.

Change #1038873 had a related patch set uploaded (by Jforrester; author: Jforrester):

[mediawiki/extensions/EventBus@master] JobQueueEventBus: Log invalid result as its own key, pushed as an object

https://gerrit.wikimedia.org/r/1038873

gerritbot added a project: Patch-For-Review.Jun 4 2024, 5:29 PM

Change #1038873 merged by jenkins-bot:

[mediawiki/extensions/EventBus@master] JobQueueEventBus: Log invalid result as its own key, pushed as an object

https://gerrit.wikimedia.org/r/1038873

Maintenance_bot removed a project: Patch-For-Review.Jun 6 2024, 4:30 AM

ReleaseTaggerBot added a project: MW-1.43-notes (1.43.0-wmf.9; 2024-06-11).Jun 6 2024, 5:00 AM

Change #1096667 had a related patch set uploaded (by Paladox; author: Jforrester):

[mediawiki/extensions/EventBus@REL1_42] JobQueueEventBus: Log invalid result as its own key, pushed as an object

https://gerrit.wikimedia.org/r/1096667

gerritbot added a project: Patch-For-Review.Nov 24 2024, 8:55 PM

Change #1096667 merged by jenkins-bot:

[mediawiki/extensions/EventBus@REL1_42] JobQueueEventBus: Log invalid result as its own key, pushed as an object

https://gerrit.wikimedia.org/r/1096667

Maintenance_bot removed a project: Patch-For-Review.Nov 24 2024, 9:30 PM

Ahoelzl moved this task from Event Platform Backlog to Backlog on the Data-Engineering board.Jan 18 2025, 12:06 AM

Adding Wikimedia-production-error, as per that tag’s description it can also be used to track Beta Cluster errors

Restricted Application changed the subtype of this task from "Task" to "Production Error". · View Herald TranscriptMar 21 2025, 5:39 AM

Reedy moved this task from Untriaged to Mar 2025 on the Wikimedia-production-error board.Mar 21 2025, 10:17 PM

Krinkle moved this task from Mar 2025 to Jan-Mar 2025 on the Wikimedia-production-error board.Apr 1 2025, 2:05 AM

Ottomata moved this task from Backlog to Radar on the Event-Platform board.Sep 17 2025, 3:04 PM

cirrusSearchCheckerJob JobQueueErrors (Could not enqueue jobs) on Beta ClusterOpen, Needs TriagePublicPRODUCTION ERRORActions

Description

Details

Related Objects

Event Timeline

cirrusSearchCheckerJob JobQueueErrors (Could not enqueue jobs) on Beta Cluster
Open, Needs TriagePublicPRODUCTION ERROR
Actions