Failed executing job: cirrusSearchElasticaWrite [...]
Closed, ResolvedPublicPRODUCTION ERROR
Actions

Assigned To

Authored By

	MarcoAurelio
	Nov 6 2019, 5:58 PM

Description

Error

MediaWiki version: 1.35.0-alpha

message

Failed executing job: cirrusSearchElasticaWrite Special: method=sendData arguments=["content","array(...)"] serialized=1 cluster=default doNotRetry= createdAt=1573062797 errorCount=4 retryCount=0 requestId=XcMIiawQAI0AABM@mE0AAAAW jobReleaseTimestamp=1573063053

Impact

https://kibana4.wmflabs.org/goto/b6642e8841413ab55d8f53973d6ed66c

Notes

Details

Request ID: XcMIiawQAI0AABM@mE0AAAAW
Request URL: n/a

Stack Trace

exception.trace

n/a

	Subject	Repo	Branch	Lines +/-
	[cirrus] Explicitly choose the search clusters to write to in beta cluster	operations/mediawiki-config	master	+1 -1
	Enable CirrusSearch log channel on beta cluster	operations/mediawiki-config	master	+1 -0

Customize query in gerrit

Related Objects

Mentioned In: T238883: wgCirrusSearchClusters should not merge arrays

Event Timeline

MarcoAurelio created this task.Nov 6 2019, 5:58 PM

Restricted Application added a project: Discovery-Search. · View Herald TranscriptNov 6 2019, 5:58 PM

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

Not (yet) seen in production.

For whatever reason beta cluster isn't logging any of the CirrusSearch channels that production does, so there is no information about what actually went wrong. Will need to fix beta cluster logging first.

EBernhardson claimed this task.Nov 21 2019, 8:21 PM

EBernhardson triaged this task as High priority.

EBernhardson moved this task from needs triage to Current work on the Discovery-Search board.

EBernhardson edited projects, added Discovery-Search (Current work); removed Discovery-Search.

Change 552332 had a related patch set uploaded (by EBernhardson; owner: EBernhardson):
[operations/mediawiki-config@master] Enable CirrusSearch log channel on beta cluster

https://gerrit.wikimedia.org/r/552332

Change 552332 merged by jenkins-bot:
[operations/mediawiki-config@master] Enable CirrusSearch log channel on beta cluster

https://gerrit.wikimedia.org/r/552332

Maintenance_bot removed a project: Patch-For-Review.Nov 21 2019, 9:10 PM

The error message is: Couldn't connect to host, Elasticsearch down?

This looks to be because there are two clusters defined, default and eqiad. There should only be one, and only one is defined in InitialiseSettings.php. Checked prod and it also has the spurious default cluster. No error occurs on production because $wgCirrusSearchWriteClusters is set to specific clusters, as opposed to beta which sets it to null which accepts all defined clusters.

Likely this is some problem with the extension.json transition that wasn't noticed till now. It should not be adding extra servers into the list.

Change 552352 had a related patch set uploaded (by EBernhardson; owner: EBernhardson):
[operations/mediawiki-config@master] [cirrus] Explicitly choose the search clusters to write to in beta cluster

https://gerrit.wikimedia.org/r/552352

gerritbot added a project: Patch-For-Review.Nov 21 2019, 11:02 PM

Change 552352 merged by jenkins-bot:
[operations/mediawiki-config@master] [cirrus] Explicitly choose the search clusters to write to in beta cluster

https://gerrit.wikimedia.org/r/552352

Maintenance_bot removed a project: Patch-For-Review.Nov 21 2019, 11:10 PM

EBernhardson mentioned this in T238883: wgCirrusSearchClusters should not merge arrays.Nov 21 2019, 11:40 PM

EBernhardson moved this task from Incoming to Needs Reporting on the Discovery-Search (Current work) board.Nov 22 2019, 12:03 AM

TJones closed this task as Resolved.Nov 27 2019, 4:41 PM