RuntimeException: Received cirrusSearchElasticaWrite job for an unwritable cluster cloudelastic.
Closed, ResolvedPublicPRODUCTION ERROR
Actions

Assigned To

Authored By

	dancy
	Feb 15 2024, 6:25 PM

Description

Error

mwversion: 1.42.0-wmf.18
reqId: efc87c8746cb657b635c2ad4
Find reqId in Logstash

normalized_message

[{reqId}] {exception_url}   RuntimeException: Received cirrusSearchElasticaWrite job for an unwritable cluster cloudelastic.

exception.trace

from /srv/mediawiki/php-1.42.0-wmf.18/extensions/CirrusSearch/includes/Job/JobTraits.php(92)
#0 /srv/mediawiki/php-1.42.0-wmf.18/extensions/CirrusSearch/includes/Job/ElasticaWrite.php(151): CirrusSearch\Job\CirrusGenericJob->decideClusters()
#1 /srv/mediawiki/php-1.42.0-wmf.18/extensions/CirrusSearch/includes/Job/JobTraits.php(144): CirrusSearch\Job\ElasticaWrite->doJob()
#2 /srv/mediawiki/php-1.42.0-wmf.18/extensions/EventBus/includes/JobExecutor.php(80): CirrusSearch\Job\CirrusGenericJob->run()
#3 /srv/mediawiki/rpc/RunSingleJob.php(60): MediaWiki\Extension\EventBus\JobExecutor->execute(array)
#4 {main}

Notes

30816 of these in the last 15 minutes. They have been arriving in big bursts since around Feb 8, 2024.

Details

Request URL: https://mw-jobrunner.discovery.wmnet/rpc/RunSingleJob.php

Related Objects

Mentioned Here: T356526: High level of backend errors for CirrusSearch jobs in jobrunners
T355617: Migrate cloudelastic from public to private IPs
T349796: Move MediaWiki jobs to mw-on-k8s

Event Timeline

dancy created this task.Feb 15 2024, 6:25 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 15 2024, 6:25 PM

dancy added a project: CirrusSearch.Feb 15 2024, 6:28 PM

Restricted Application added a project: Discovery-Search. · View Herald TranscriptFeb 15 2024, 6:28 PM

I note these are all coming from kube-mw-jobrunners

Not hard to imagine a scenario where this interferes with canaries during a deployment.

I think this is fallout of T349796: Move MediaWiki jobs to mw-on-k8s?

In T357713#9548631, @brennen wrote:

Not hard to imagine a scenario where this interferes with canaries during a deployment.

I think this is fallout of T349796: Move MediaWiki jobs to mw-on-k8s?

hrm, seems to have started Feb 8. Which doesn't align, quite, with actions on that task, but tagging in serviceops to take a confirm.

Going back to IRC logs from a bit ago, what I had in mind was T355617: Migrate cloudelastic from public to private IPs.

931134 2024-02-06 12:13:07     dancy   Many `.17 e/C/i/J/JobTraits:92  Received cirrusSearchElasticaWrite job for an unwritable cluster cloudelastic.`
931136 2024-02-06 12:13:14     dancy   brennen:
931137 2024-02-06 12:13:26     dancy   although they're trailing off.
931138 2024-02-06 12:13:53     inflatador      dancy what was the timeline for those errors? We've been migrating cloudelastic to private IPs
931139 2024-02-06 12:13:53     dancy   and now gone. :-)
931140 2024-02-06 12:14:07     dancy   I was looking at last 15 minutes..  and then they went away
931141 2024-02-06 12:14:41     dancy   inflatador: Thanks for the info!
931142 2024-02-06 12:14:54     dancy   Crisis averted. :-)
931143 2024-02-06 12:15:15     inflatador      sure, more context in T355617 if interested. No impact expected, but that cluster doesn't have a lot of redundancy ;(

In T357713#9548678, @brennen wrote:

Going back to IRC logs from a bit ago, what I had in mind was T355617: Migrate cloudelastic from public to private IPs.

Ah ha, that ticket does align with the timing on this error message. Shifting tags again (sorry serviceops), tagging Data-Platform-SRE

For completeness, this is T356526: High level of backend errors for CirrusSearch jobs in jobrunners and seems to have been resolved by https://gerrit.wikimedia.org/r/1003827

In T357713#9549879, @Clement_Goubert wrote:

For completeness, this is T356526: High level of backend errors for CirrusSearch jobs in jobrunners and seems to have been resolved by https://gerrit.wikimedia.org/r/1003827

Thanks!

	F41915620: T357713.png
	Feb 15 2024, 6:27 PM

RuntimeException: Received cirrusSearchElasticaWrite job for an unwritable cluster cloudelastic.Closed, ResolvedPublicPRODUCTION ERRORActions