Problem
Predictor pods on the MI300 nodes declare no ephemeral-storage request or limit. The scheduler therefore can't account for disk when placing pods, so it binds pods onto nodes that can't physically hold the image + model download. The result is eviction loops that eventually taint the node disk-pressure.
Initial info and suggestion in the comment: https://phabricator.wikimedia.org/T431017#12083858
Observed on ml-serve1012: repeated evictions across namespaces (llm/cope-b-a4b, and qwen36-27b during a full-GPU test), each killed mid-download. Every container shows request is 0:
Reason: Evicted Message: The node was low on resource: ephemeral-storage. Threshold quantity: 14650510349, available: 12034436Ki. Container kserve-container was using 317968Ki, request is 0, has larger consumption of ephemeral-storage.
Once tainted, the node also blocks new scheduling:
0/17 nodes are available: 1 node(s) had untolerated taint {node.kubernetes.io/disk-pressure: }, ...Proposed fix
Declare ephemeral-storage requests and limits on the predictor containers, sized to the model weights (e.g. cope-b-a4b ~52G BF16, qwen3-14b ~28G, qwen36-27b-FP8 ~29G).
With requests set:
- the scheduler sees disk and leaves an unfittable pod Pending with a clear message, instead of binding then eviction-looping;
- the limit caps the download, so an oversized model fails only its own pod rather than pressuring the whole node.
This is a values-only change in helmfile.d/ml-services/ — the kserve-inference chart passes container resources through verbatim. No chart or code change needed.
Scope
Apply to the LLM predictors on the MI300 nodes (llm/, experimental/). Size each per its model.