Elasticsearch stores some data directly in the java heap.
We should monitor this usage because it can directly affect the gc.
The data we could collect is available via the node stats api:
| Subject | Repo | Branch | Lines +/- | |
|---|---|---|---|---|
| Monitor usage of in-memory elasticsearch datastructures | operations/puppet | production | +15 -0 |
This is a task is a good candidate to be tackled by someone without full knowledge of the details of how we use elasticsearch. This is a simple adaptation of the wmfelastic.py script.
Is indices.request_cache truly needed? A quick check with curl localhost:9200/_nodes/stats?groups=_all | jq '.nodes | to_entries | map({ key: .value.name, value: .value.indices.request_cache}) | from_entries' shows they are 0 across the board (on both the search and logstash prod clusters). Incoming patch covers the other items, but not this one.
Change 311848 had a related patch set uploaded (by EBernhardson):
Monitor usage of in-memory elasticsearch datastructures
Change 311848 merged by Gehel:
Monitor usage of in-memory elasticsearch datastructures
Code deployed, metrics are visible in Graphite for elastic1020 (other should follow once puppet runs).