Create Icinga check for failed shard allocation
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	• Mathew.onipe
	Jan 3 2019, 12:38 PM

Description

Following https://grafana.wikimedia.org/d/000000455/elasticsearch-percentiles-prometheus?panelId=64&fullscreen&orgId=1&from=now-30d&to=now&var-cluster=eqiad&var-smoothing=1&var-exported_cluster=search&edit, we discovered a shard was unassigned since 02/12/2018.

We should have icinga alert us if there is any case as such.
Querying /_cluster/allocation/explain should give us what we need. This check should happen at low frequency. I'm proposing:
Freq: Daily or 24h
retry: 1h
tries: 3

Details

Subject	Repo	Branch	Lines +/-
elasticsearch: fixed duplicated check description	operations/puppet	production	+4 -4
icinga: enable check for logstash	operations/puppet	production	+22 -0
elasticsearch: unassigned shard icinga check	operations/puppet	production	+10 -0
icinga: enable check for psi and omega clusters	operations/puppet	production	+65 -82
icinga: enable check for psi and omega clusters	operations/puppet	production	+51 -84
icinga: enable check for psi and omega clusters	operations/puppet	production	+47 -80
Elasticsearch failed shard allocation check	operations/puppet	production	+177 -1

Customize query in gerrit

Related Objects

Duplicates Merged Here: T215451: Logging elasticsearch had shards unassigned and icinga didn't mention it

Event Timeline

• Mathew.onipe created this task.Jan 3 2019, 12:38 PM

Restricted Application edited projects, added Discovery-Search; removed Discovery-Search (Current work). · View Herald TranscriptJan 3 2019, 12:38 PM

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

• Mathew.onipe triaged this task as Medium priority.Jan 3 2019, 12:38 PM

• Mathew.onipe edited projects, added Discovery-Search (Current work); removed Discovery-Search.

• Mathew.onipe moved this task from Incoming to not in use - please delete on the Discovery-Search (Current work) board.Jan 4 2019, 8:20 AM

Change 482297 had a related patch set uploaded (by Mathew.onipe; owner: Mathew.onipe):
[operations/puppet@production] Elasticsearch failed shard allocation check

https://gerrit.wikimedia.org/r/482297

gerritbot added a project: Patch-For-Review.Jan 4 2019, 1:11 PM

Change 482297 merged by Gehel:
[operations/puppet@production] Elasticsearch failed shard allocation check

https://gerrit.wikimedia.org/r/482297

This check has been deployed for the main cirrus clusters (eqiad+codfw).

We still need to add it for :

psi / omega cirrus clusters
relforge
logstash

Change 484679 had a related patch set uploaded (by Mathew.onipe; owner: Mathew.onipe):
[operations/puppet@production] icinga: enable check for psi and omega cluster

https://gerrit.wikimedia.org/r/484679

Change 484685 had a related patch set uploaded (by Mathew.onipe; owner: Mathew.onipe):
[operations/puppet@production] icinga: enable check for logstash

https://gerrit.wikimedia.org/r/484685

• Mathew.onipe removed a project: Patch-For-Review.Jan 22 2019, 6:52 PM

• Phabricator_maintenance moved this task from Backlog to Acknowledged on the SRE board.Jan 26 2019, 10:58 PM

debt moved this task from not in use - please delete to Incoming on the Discovery-Search (Current work) board.Jan 31 2019, 10:54 PM

Change 484679 merged by Gehel:
[operations/puppet@production] icinga: enable check for psi and omega clusters

https://gerrit.wikimedia.org/r/484679

Change 488453 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/puppet@production] elasticsearch: fixed duplicated check description

https://gerrit.wikimedia.org/r/488453

gerritbot added a project: Patch-For-Review.Feb 6 2019, 3:00 PM

Change 488485 had a related patch set uploaded (by Mathew.onipe; owner: Mathew.onipe):
[operations/puppet@production] icinga: enable check for psi and omega clusters

https://gerrit.wikimedia.org/r/488485

fgiunchedi awarded a token.Feb 7 2019, 9:23 AM

fgiunchedi merged a task: T215451: Logging elasticsearch had shards unassigned and icinga didn't mention it.Feb 7 2019, 9:42 AM

fgiunchedi subscribed.

Change 488485 merged by Gehel:
[operations/puppet@production] icinga: enable check for psi and omega clusters

https://gerrit.wikimedia.org/r/488485

Change 489154 had a related patch set uploaded (by Mathew.onipe; owner: Mathew.onipe):
[operations/puppet@production] icinga: enable check for psi and omega clusters

https://gerrit.wikimedia.org/r/489154

Change 489154 merged by Gehel:
[operations/puppet@production] icinga: enable check for psi and omega clusters

https://gerrit.wikimedia.org/r/489154

Change 489765 had a related patch set uploaded (by Mathew.onipe; owner: Mathew.onipe):
[operations/puppet@production] elasticsearch: unassigned shard icinga check

https://gerrit.wikimedia.org/r/489765

Change 489765 merged by Gehel:
[operations/puppet@production] elasticsearch: unassigned shard icinga check

https://gerrit.wikimedia.org/r/489765

• Mathew.onipe moved this task from Incoming to Needs Reporting on the Discovery-Search (Current work) board.Feb 12 2019, 10:59 AM

Change 484685 abandoned by Mathew.onipe:
icinga: enable check for logstash

Reason:
was merged here: https://gerrit.wikimedia.org/r/c/operations/puppet/ /489765

https://gerrit.wikimedia.org/r/484685

debt closed this task as Resolved.Feb 15 2019, 6:58 PM

Change 488453 abandoned by Gehel:
elasticsearch: fixed duplicated check description

Reason:
already refactored

https://gerrit.wikimedia.org/r/488453