Related T404726: [tools,infra,k8s] scale up the cluster, specifically CPU

Memory is tricky in that if your job hits the limit, or the host has no more free memory (if we overcommited and more than one job uses more than their request) then the pod is killed, and most tools (specially scheduledjobs) are not designed with that in mind.

I see some ways forward (just ideas, not exclusive):

• Reduce the default request
• Increase the number of workers (next ones should have 2x the mem iirc)
• Find a better signal to alert on
  • Iirc, something we talked about was using the number of pods in pending state (as in, waiting for allocation, this helps with overall cluster usage, on the reservation side only though)
  • Another signal might be counting the number of pods killed by OOM (ex. to notice pods that have memory peaks, this would give an idea on the actual usage/limit side)