logstash1002 hit an OOM today and then fell out of the cluster. In elasticsearch 2.3 aggregations are done on the machine running the queries, which is always 1001-3, these might be a bit more memory hungry than facets on 1.7 were. The heap is currently set to 2G and 1001-3 all have 5G+ unused (completely, in addition to a 4-5G disk cache). We can likely double the amount of memory available to these instances without any concern.
Description
Description
Details
Details
| Subject | Repo | Branch | Lines +/- | |
|---|---|---|---|---|
| Increase elasticsearch heap on logstash routing instances | operations/puppet | production | +1 -1 |
Related Objects
Related Objects
Event Timeline
Comment Actions
Change 300440 had a related patch set uploaded (by EBernhardson):
Increase elasticsearch heap on logstash routing instances
Comment Actions
Change 300440 merged by Gehel:
Increase elasticsearch heap on logstash routing instances
Comment Actions
Mentioned in SAL [2016-07-22T14:21:23Z] <gehel> rolling restart of logstash100[1-3] - T141063
Comment Actions
Increase of memory deployed, elasticsearch restarted on logstash100[1-3], checked that new heap is 4Go.