Why don't we just fix the ganglia plugin we're already using? Right now Elasticsearch plugins are difficult because any code changes require a rolling restart of the whole elasticsearch cluster. But the ganglia plugin just need bouncing gmond.

All and all I think a monitoring Elasticsearch plugin is a bad idea for us - monitor it in some external application. Once we get faster rolling restarts I won't mind, but that is at least several months away.

The benefit of statsd is that, through graphite, we can run more powerful/complicated queries across the data. For example individual indexes are reported on and you can generate a dashboard that sum's stats across all indexes, or just ones that match a particular globbing pattern. I have to agree though that as long as the 36 hour restart is in place we shouldn't look at this too closely. Adding pausing of indexing as a blocking task.

this should probably be pulled from the sprint, its blocked on search services taking too long to restart. those tasks need to be resolved first.

Change 223202 had a related patch set uploaded (by EBernhardson):
Add statsd reporting plugin

https://gerrit.wikimedia.org/r/223202

Installed to elastic cluster and did a rolling restart. Everything looks to be collecting now, stats are found at

http://graphite.wmflabs.org in the elasticsearch.beta-search namespace.

Example graph