While we (SRE) made some strides improving stat host experience in T373446 , we have not addressed resource contention. As a result, a runaway user-created process can still hang the machine.
Creating this ticket to link individual tasks which advance the goal of stabilizing the stat hosts.