Page MenuHomePhabricator

Create a grafana dashboard for logstash*.eqiad.wmnet based on search dashboards
Open, MediumPublic

Description

Logstash elasticsearch instances should be (if not turn it on) collecting data through diamond allowing to reuse some of the same graphs we have built for search. Set those up to give more insight into how the cluster is performing

Event Timeline

Restricted Application added subscribers: Zppix, Aklapper. · View Herald TranscriptJul 13 2016, 6:17 PM
debt triaged this task as Medium priority.Jul 14 2016, 10:22 PM
debt moved this task from needs triage to This Quarter on the Discovery-Search board.
debt added a subscriber: debt.

Let's put this new dashboard somewhere here: https://grafana.wikimedia.org/dashboard/db/elasticsearch

@EBernhardson has this been done eventually and shows up in dashboards ?

Nope, this was never done. One difficulty is we don't have strictly elasticsearch graphs, they typically have relevant data logged by CirrusSearch mixed in. A high level overview of our dashboards:

Elasticsearch Indexing: https://grafana.wikimedia.org/d/JLK3I_siz/elasticsearch-indexing?orgId=1

  • About half elasticsearch, half cirrussearch. Reports percentiles, disk activity, indexing activity, and stats about the updates we send.

Elasticsearch memory: https://grafana.wikimedia.org/d/000000462/elasticsearch-memory?orgId=1

  • Entirely elasticsearch afaict
  • Breaks out a variety of things that use noticeable amounts of memory

Elasticsearch node comparison: https://grafana.wikimedia.org/d/000000460/elasticsearch-node-comparison?orgId=1

  • Entirely elasticsearch afaict
  • Reports a variety of per-node metrics, primarily used to compare a misbehaving node to other nodes in the cluster.

In theory the last two shouldn't be too hard to modify to report logstash as well, as long as the metrics reporter is installed.