Page MenuHomePhabricator

Set ephemeral-storage requests/limits on ML predictor containers to prevent node disk eviction
Open, Needs TriagePublic

Description

Problem

Predictor pods on the MI300 nodes declare no ephemeral-storage request or limit. The scheduler therefore can't account for disk when placing pods, so it binds pods onto nodes that can't physically hold the image + model download. The result is eviction loops that eventually taint the node disk-pressure.

Initial info and suggestion in the comment: https://phabricator.wikimedia.org/T431017#12083858

Observed on ml-serve1012: repeated evictions across namespaces (llm/cope-b-a4b, and qwen36-27b during a full-GPU test), each killed mid-download. Every container shows request is 0:

Reason:   Evicted
Message:  The node was low on resource: ephemeral-storage. Threshold quantity: 14650510349, available: 12034436Ki. Container kserve-container was using 317968Ki, request is 0, has larger consumption of ephemeral-storage.

Once tainted, the node also blocks new scheduling:

0/17 nodes are available: 1 node(s) had untolerated taint {node.kubernetes.io/disk-pressure: }, ...

Proposed fix

Declare ephemeral-storage requests and limits on the predictor containers, sized to the model weights (e.g. cope-b-a4b ~52G BF16, qwen3-14b ~28G, qwen36-27b-FP8 ~29G).

With requests set:

  • the scheduler sees disk and leaves an unfittable pod Pending with a clear message, instead of binding then eviction-looping;
  • the limit caps the download, so an oversized model fails only its own pod rather than pressuring the whole node.

This is a values-only change in helmfile.d/ml-services/ — the kserve-inference chart passes container resources through verbatim. No chart or code change needed.

Scope

Apply to the LLM predictors on the MI300 nodes (llm/, experimental/). Size each per its model.

Event Timeline

Change #1308563 had a related patch set uploaded (by Bartosz Wójtowicz; author: Bartosz Wójtowicz):

[operations/deployment-charts@master] ml-services: declare ephemeral-storage for all MI300 LLM predictors

https://gerrit.wikimedia.org/r/1308563

Change #1308618 had a related patch set uploaded (by Bartosz Wójtowicz; author: Bartosz Wójtowicz):

[operations/deployment-charts@master] ml-services: declare ephemeral-storage for all MI300 LLM predictors

https://gerrit.wikimedia.org/r/1308618

Change #1308618 abandoned by Bartosz Wójtowicz:

[operations/deployment-charts@master] ml-services: declare ephemeral-storage for all MI300 LLM predictors

Reason:

Duplicate patch by mistake

https://gerrit.wikimedia.org/r/1308618

Change #1308563 merged by jenkins-bot:

[operations/deployment-charts@master] ml-services: declare ephemeral-storage for all MI300 LLM predictors

https://gerrit.wikimedia.org/r/1308563

Change #1309172 had a related patch set uploaded (by Bartosz Wójtowicz; author: Bartosz Wójtowicz):

[operations/deployment-charts@master] ml-services: Update ephemeral storage for gpt-oss.

https://gerrit.wikimedia.org/r/1309172

Change #1309172 merged by jenkins-bot:

[operations/deployment-charts@master] ml-services: Update ephemeral storage for gpt-oss.

https://gerrit.wikimedia.org/r/1309172

I've done first batch of updates targeting our LLM models, each of them now is now defining both requests and limits for ephemeral storage. All models are redeployed.

As next steps, we will update all remaining ISVCs to define requests/limits. Once this is done, we can add enforcement on chart level, making sure each InferenceService defines the ephemeral storage requests/limits.

Change #1320919 had a related patch set uploaded (by Bartosz Wójtowicz; author: Bartosz Wójtowicz):

[operations/deployment-charts@master] ml-services: declare ephemeral-storage for all ml-staging isvcs

https://gerrit.wikimedia.org/r/1320919

Change #1320919 merged by jenkins-bot:

[operations/deployment-charts@master] ml-services: declare ephemeral-storage for all ml-staging isvcs

https://gerrit.wikimedia.org/r/1320919

Change #1321883 had a related patch set uploaded (by Bartosz Wójtowicz; author: Bartosz Wójtowicz):

[operations/deployment-charts@master] ml-services: Remove revertrisk-multilingual from experimental staging.

https://gerrit.wikimedia.org/r/1321883

Change #1321883 merged by jenkins-bot:

[operations/deployment-charts@master] ml-services: Remove revertrisk-multilingual from experimental staging.

https://gerrit.wikimedia.org/r/1321883

Change #1321915 had a related patch set uploaded (by Bartosz Wójtowicz; author: Bartosz Wójtowicz):

[operations/deployment-charts@master] ml-services: Declare ephemeral-storage on prod services.

https://gerrit.wikimedia.org/r/1321915

Change #1321915 merged by jenkins-bot:

[operations/deployment-charts@master] ml-services: Declare ephemeral-storage on prod services.

https://gerrit.wikimedia.org/r/1321915

Change #1321967 had a related patch set uploaded (by Bartosz Wójtowicz; author: Bartosz Wójtowicz):

[operations/deployment-charts@master] ml-services: Lower ephemeral-storage requests for revscoring services.

https://gerrit.wikimedia.org/r/1321967

Change #1321967 merged by jenkins-bot:

[operations/deployment-charts@master] ml-services: Lower ephemeral-storage requests for revscoring services.

https://gerrit.wikimedia.org/r/1321967

All KServe deployments on staging and production are now re-deployed with ephemeral-storage request and limits declared. During the work on this task I also noticed that we have some deployments not declaring memory and cpu requests/limits either, which can be tackled in a follow up ticket. Once this will be done, we could add requirements for all cpu/memory/storage be required on the chart level.

The remaining services not having ephemeral-storage are liftwing-openapi-server, ores-legacy, recommendation-api-ng and tts-section-generator, those will be updated soon as well.