Determine cause/fix cloudelastic cross-cluster search missing index errors
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	bking
	Feb 23 2024, 9:20 PM

Description

While investigating T355795 ,@EBernhardson noticed noticed that a number of errors in Logstash related to missing indices .

Creating this ticket to investigate these cross-cluster search missing index errors (and hopefully fix them).

Details

	Subject	Repo	Branch	Lines +/-
	Elastic: unique systemd timer name per cluster	operations/puppet	production	+1 -1
	elasticsearch: prevent cross cluster seed config drift	operations/puppet	production	+68 -0

Customize query in gerrit

Related Objects

Mentioned Here: T358802: Cloudelastic: alert on inconsistencies between running CCS running state and active master-eligibles
T355795: Fix "requests triggering circuit breakers" Elastic alert

Event Timeline

bking created this task.Feb 23 2024, 9:20 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 23 2024, 9:20 PM

Gehel triaged this task as High priority.Feb 26 2024, 2:18 PM

Gehel moved this task from Incoming to 2024.02.12 - 2024.03.03 on the Data-Platform-SRE board.

Gehel edited projects, added Data-Platform-SRE (2024.02.12 - 2024.03.03); removed Data-Platform-SRE.

Gehel moved this task from needs triage to Current work on the Discovery-Search board.Feb 26 2024, 2:20 PM

Gehel edited projects, added Discovery-Search (Current work); removed Discovery-Search.

Gehel edited projects, added Data-Platform-SRE (2024.03.04 - 2024.03.24); removed Data-Platform-SRE (2024.02.12 - 2024.03.03).Mar 4 2024, 1:34 PM

Gehel moved this task from Incoming to DPE-SRE on the Discovery-Search (Current work) board.Mar 4 2024, 4:42 PM

bking renamed this task from Determine cause/fix cross-cluster search missing index errors to Determine cause/fix cloudelastic cross-cluster search missing index errors.Mar 4 2024, 4:42 PM

Gehel moved this task from Backlog to In Progress on the Data-Platform-SRE (2024.03.04 - 2024.03.24) board.Mar 22 2024, 8:46 AM

Gehel edited projects, added Data-Platform-SRE (2024.03.25 - 2024.04.14); removed Data-Platform-SRE (2024.03.04 - 2024.03.24).Mar 22 2024, 8:48 AM

Is this related to T358802 ?

Gehel moved this task from Backlog to Needs Review on the Data-Platform-SRE (2024.03.25 - 2024.04.14) board.Mar 22 2024, 9:09 AM

Gehel moved this task from DPE-SRE to Needs Reporting on the Discovery-Search (Current work) board.Mar 25 2024, 4:26 PM

This is related to the same cross cluster setting configuration issue.

Gehel moved this task from Needs Review to Done on the Data-Platform-SRE (2024.03.25 - 2024.04.14) board.Apr 2 2024, 3:23 PM

Reopening, as we could fix this for good by rendering the YAML file in this puppet plan as json and writing it to the cluster settings periodically (such as with a systemd timer).

bking reopened this task as In Progress.Apr 9 2024, 9:48 PM

bking claimed this task.

bking lowered the priority of this task from High to Medium.

bking moved this task from Done to In Progress on the Data-Platform-SRE (2024.03.25 - 2024.04.14) board.

Change #1018360 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] WIP: elasticsearch: prevent cross cluster seed config drift

https://gerrit.wikimedia.org/r/1018360

gerritbot added a project: Patch-For-Review.Apr 9 2024, 9:51 PM

Gehel edited projects, added Data-Platform-SRE (2024.04.15 - 2024.05.05); removed Data-Platform-SRE (2024.03.25 - 2024.04.14).Apr 15 2024, 12:39 PM

Gehel moved this task from Backlog to In Progress on the Data-Platform-SRE (2024.04.15 - 2024.05.05) board.

Change #1018360 merged by Bking:

[operations/puppet@production] elasticsearch: prevent cross cluster seed config drift

https://gerrit.wikimedia.org/r/1018360

Change #1020326 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] Elastic: unique systemd timer name per cluster

https://gerrit.wikimedia.org/r/1020326

Change #1020326 merged by Bking: