Adding SRE for triage (not sure who maintains the service that caused the warning)

The redis logs don't have anything really interesting in them. I restarted redis on arclamp1001 as a first step to see if these errors clear up. I will circle back on the logs later this week to see if this issue persists.

I think the hosts are periodically running out of memory. The graphs are sometimes noisier/less obvious than other times, but e.g. this time period in Grafana shows what looks to me like a clear memory problem that overwhelms the host to the point of suddenly not responding to Prometheus metric collection:

I do not know what makes the memory spikes happen, but I'm quite convinced that it's worth looking at this angle.

The restart doesn't appear to have alleviated the problem - I see more redis connection exceptions since the restart.

Chances are it's hitting out of memory, as you say, due to flamegraph generation. Since this is a physical host, memory upgrades will have to be coordinated with dc-ops.

Memory was upgraded on arclamp1001 today and I'm seeing used memory max out at around 32GB. I'll keep an eye on it.

In T432730#12141549, @Dreamy_Jazz wrote:

Adding SRE for triage (not sure who maintains the service that caused the warning)

For future routing of the issues, I added ownership information to Redis clusters Wikitech page