Search backend error during sending 1 documents to the commonswiki_file_ [...] primary shard is not active
Closed, DuplicatePublicPRODUCTION ERROR
Actions

Assigned To

None

Authored By

	dduvall
	Jun 8 2022, 6:42 PM

Description

Error

mwversion: undefined
reqId: undefined
Find requests in Logstash

normalized_message

pulled partial message from truncated json message: "Search backend error during sending 1 documents to the commonswiki_file_1647920262 index(s) after 8: bulk: Error in one or more bulk request actions:\n\nupdate: /commonswiki_file_1647920262/_doc/97462744 caused [commonswiki_file_1647920262][11] primary shard is not active Timeout: [1ms], request: [...]"

exception.trace

Impact

A large spike (~ 2000k) of these messages was noticed on the jsonTruncated channel during group1 promotion of 1.39.0-wmf.15. User impact is unclear.

Notes

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Resolved	Release	dduvall	T308068 1.39.0-wmf.15 deployment blockers
		Duplicate	PRODUCTION ERROR	None	T310216 Search backend error during sending 1 documents to the commonswiki_file_ [...] primary shard is not active

Event Timeline

dduvall triaged this task as Unbreak Now! priority.Jun 8 2022, 6:42 PM

dduvall created this task.

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 8 2022, 6:42 PM

dduvall added a parent task: T308068: 1.39.0-wmf.15 deployment blockers.Jun 8 2022, 6:42 PM

dduvall added a project: Discovery-Search.Jun 8 2022, 6:50 PM

Sorry for the sparse report. It's by virtue of coming through on the truncated channel it seems. I'll update the logstash link to at least point to channel:"jsonTruncated" during the train time window.

dduvall updated the task description. (Show Details)Jun 8 2022, 6:53 PM

This is the same index as T309648: Restore lost index in cloudelastic

bking added a subscriber: bking.Jun 8 2022, 7:18 PM

Agusbou2015 added a subscriber: Agusbou2015.Jun 8 2022, 11:10 PM

thcipriani moved this task from Untriaged to Jun 2022 on the Wikimedia-production-error board.Jun 9 2022, 4:08 PM

dduvall closed this task as a duplicate of T309648: Restore lost index in cloudelastic.Jun 9 2022, 5:26 PM