Cloudelastic has started to alert more regularly over the last month first on march 3rd, then nothing for awhile, and now alerts on the 23rd, 24th, 29th, 30th and the 1st. The alerts come from psi and omega, the small jvms. In particular looking at the graphs we can see that just before the beginning of april we started running out of memory. The old pool filled up and the survivor pool hasn't been used since. The young pool retained enough working space to continue operation, but it wont be sustainable.
Plausibly we should either reduce memory usage (T279009 would likely help, cut index/shard counts in half) or increase jvm heap size.