In IRC, @EBernhardson mentioned that:
Since high load on elastic2044 was a contributing factor to a recent Elasticsearch outage, creating this ticket to:
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | Gehel | T346945 2023-09-20 Elasticsearch unavailable incident | |||
| Resolved | bking | T347338 Investigate performance differences between elastic2037-2054 and 2055-2086 |
@RKemper noted that the difference here is the available memory. everything < 2055 has 128G of memory, >=2055 has 256G of memory. The additional memory is doing exactly what we expected of it, reducing the need to go out to disk while serving typical query loads. It seems unlikely we will be able to do anything, it looks like this is a fundamental difference in hardware.
Upon further review, it looks like we have confirmed the reason for the performance differences. I don't think we need to change anything (such as tiering hosts) so I'm going to close this out. Thanks everyone for taking a look!