In T345462 there was a partial outage where new servers could not be launched. This was due to all the memory in the worker nodes having been assigned. However the actual usage was rather lower than the requested usage.

At the time there were 87 single user containers running, of which the mean memory usage was 390Mi, and the median was 264Mi. Currently each server is given a 1Gi request and a 3Gi limit. At the time of T345462 only 2 of the 87 containers were using more than the 1Gi request (Using 1388Mi and 1395Mi). The memory usage is at about 50% (50%, 48% and 51% at the time). In T345462 the number of worker nodes was increased to resolve the issue.

Though 50% utilization is pretty low, is it reasonable to reduce the memory request (not limit) of the single user containers. In practice if the memory request were lower, no one would have observed any difference and the users who could not login, would have been able to. As there was plenty of unused memory about.

Of the 87 servers, 23 were using more than 500Mi and 13 were using more than 750Mi. Perhaps lowering the threshold to one of these figures is advisable, at least until the nodes get closer to 70% usage.