These graphs used to work, but they now show as completely flat lines. Querying a few prometheus reporters on the elasticsearch instances directly shows the underlying values are not 0. Somewhere between where the values are generated and these dashboards something changed, find out what and repair.

In particular the "full_text ema eqiad" on the elasticsearch-mjolnir-msearch[1] board is a moving average of the full_text search load. It reports 0 for both DC's, but instances correctly report a few thousand qps.

ebernhardson@elastic1050:~$ curl localhost:9171 | grep ema
# HELP mjolnir_msearch_ema_qps Local estimate of canary index qps
# TYPE mjolnir_msearch_ema_qps gauge
mjolnir_msearch_ema_qps 2597.2681008786576

[1] https://grafana.wikimedia.org/d/000000616/elasticsearch-mjolnir-msearch