@Gehel How do you recommend prioritising this?

I'd say this is fairly low priority. Elasticsearch has good monitoring, it is not urgent to improve it significantly. I'd like keep that task for my next "kitchen hackathon" as it seems to be a good introduction to our infrastructure.

Keeping track of curl elastic1030.eqiad.wmnet:9200/_cluster/stats | jq .indices.completion.size_in_bytes could help diagnose issue with completion suggester rebuild.

Raising this task's priority:

Getting a storm of alerts like we get now on cluster failures is distracting, especially when the failure is sourced at other major events at the datacenter (such as network maintenance like today, or random network failures) that also need troubleshooting/debugging. We've seen it twice now during the cr1-eqiad/cr2-eqiad JunOS upgrades and, fortunately, this was planned work so we were kind of around and aware of what the root cause might have been.

Moreover, if we had one service check like the task above suggests, we could also make it paging, thus improving our response times (such as in weekends and such).

Change 305519 had a related patch set uploaded (by Gehel):
Elasticsearch - check shards via the service, not via each individual node

https://gerrit.wikimedia.org/r/305519

Change 305519 merged by Gehel:
elasticsearch - check shards via the service, not via each individual node

https://gerrit.wikimedia.org/r/305519

@EBernhardson does not see as many alerts any more.

@Gehel Do you consider this resolved? If not, can you give details on some of the outstanding work? Thanks!

The first part of checking cluster state against the service address (search.svc...) is done, but the check of aliases is not there yet.