Following up from T419674: ToolforgeKubernetesCapacity alert actionability

In T419674#11696855, @dcaro wrote:

Memory is tricky in that if your job hits the limit, or the host has no more free memory (if we overcommited and more than one job uses more than their request) then the pod is killed, and most tools (specially scheduledjobs) are not designed with that in mind.

I see some ways forward (just ideas, not exclusive):

• Reduce the default request

It would also be interesting I think to see how the distribution of tool memory request vs usage looks like in terms of understanding what a good default request looks like.

• Increase the number of workers (next ones should have 2x the mem iirc)

Given that memory requests creep up over time (75% a month ago, 81% now) and when we hit 100% we can't schedule new workloads, then I think in the immediate term adding more workers for headroom is the right thing to do IMHO.

• Find a better signal to alert on
  • Iirc, something we talked about was using the number of pods in pending state (as in, waiting for allocation, this helps with overall cluster usage, on the reservation side only though)
  • Another signal might be counting the number of pods killed by OOM (ex. to notice pods that have memory peaks, this would give an idea on the actual usage/limit side)

I'm definitely +1 on at least trying both of these and see how much signal we get

Something we can improve there also is that they are metrics with a big cardinality (has the pod/namespace and such), so maybe we want to aggregate them somehow at the prometheus level (that might be useful also for other reasons, like storing the count of tools for very long, and others, unrelated to this task though).