Page MenuHomePhabricator

Cloudelastic: alert on inconsistencies between running CCS running state and active master-eligibles
Closed, ResolvedPublic

Description

We changed the cloudelastic master-eligibles in T361305 , but we neglected to update the CCS (cross-cluster search) settings on the running cluster. As a result, users lost the ability to search across the cluster (see T358541 ).

We already check for this on production master-eligible nodes (Puppet code). Creating this ticket to monitor cloudelastic as well.

Event Timeline

Gehel triaged this task as Medium priority.Mar 5 2024, 10:42 AM

Change 1012703 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] cloudelastic: check/alert on cluster inconsistencies

https://gerrit.wikimedia.org/r/1012703

Just capturing a few notes from today's pairing session with @EBernhardson and @RKemper .

We think the best way to do this is to create the cross-cluster data structure on the elastic hosts as json, and have Puppet update the settings when they don't match (using onlyif conditional).

Change 1012703 merged by Bking:

[operations/puppet@production] cloudelastic: check/alert on cluster inconsistencies

https://gerrit.wikimedia.org/r/1012703

The immediate work is done, although we still need to migrate these checks away from icinga. (See T358029 and T346438.) Closing...