The deployment of qwen3.6 model might have affected this https://phabricator.wikimedia.org/T425680#12075549

I have manually deleted some pods and a llm-qwen3-14b service that was deployed recently and was schedule on the same node https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1306286

The cope-b service seems to work now, but we need to monitor in case pods are evicted due to the same reason.

(Adding the disk findings from ml-serve1012 and some context about the calico side of this)

Where the disk went:

The pressure is on /var/lib/kubelet (backs emptyDirs)

/dev/mapper/vg0-containerd  1.3T  384G  796G  33% /var/lib/containerd
/dev/mapper/vg0-kubelet     137G  107G   24G  82% /var/lib/kubelet

~82G of that is model weights in kserve-provision-location emptyDirs held by long-running predictors (28G llm/qwen3-14b, 28G experimental/qwen3-14b, 26G gpt-oss-safeguard-20b). The ~24G of headroom is less than one more model download, so each new cope-b/qwen3 pod downloads until it trips the node eviction threshold, dies, frees its partial download, and the ReplicaSet retries (~40-50 evictions/hour)

Suggested fixes:

1. Grow the kubelet LV: vg0 has ~796G unallocated, so the live fix is an online lvextend --resizefs per Monitoring/Disk space. To make it stick, also raise the 150G kubelet cap in partman/custom/kubernetes-node-containerd-efi.cfg (mapped to ml-serve101[2-5] in modules/profile/data/profile/installserver/preseed.yaml); the current recipe caps kubelet at 150G and gives containerd all remaining space, so a reimage would both revert the resize and consume today's free space
2. Declare ephemeral-storage requests and limits on the predictor containers, sized to the model weights (cope-b-a4b is ~52G BF16 per T427497, qwen3-14b ~28G). The eviction events show request is 0 for every container, so the scheduler cannot see disk at all and binds pods that physically cannot fit. With requests set, an unfittable pod stays Pending with a clear scheduling message instead of eviction-looping and tainting the node, and the limits cap the download itself so an oversized model fails only its own pod. This is a values-only change in helmfile.d/ml-services/llm/; the kserve-inference chart passes container resources through verbatim

Why the calico alert fired:

The ~1300 dead pod objects pushed calico-kube-controllers (200Mi memory limit on ml-serve) into an OOM crashloop, which was the kube-system alert.

I deleted the Failed pods and it recovered. The same thing happened June 5-16

Worth removing the calico memory limits in admin_ng/values/ml-serve.yaml per the wikikube precedent in T376976. That change replaced the limits with an alert that fires when a calico component sustains 3x its memory request for 10 minutes, and the alert already deploys to the ml-serve prometheus instances, so the safety net is in place before removing the limits here

Investigation: RAM-backed model storage & direct-to-VRAM loading

Summary of findings while looking into the disk-pressure evictions for the
Qwen3.6-27B-FP8 deployment (experimental ns, ml-serve-eqiad).

Root cause framing

Extending the filesystem is only a temporary fix. The model artifacts don't
need to persist on disk at all, and the extendable headroom is limited (~357G),
so larger models will hit this same wall again. Worth solving structurally
rather than just growing the FS.

Option A: RAM-backed volume for /mnt/models — not directly possible

The idea was to swap the model download directory for a medium: Memory
emptyDir (ramdisk). This does not work with the standard KServe flow:

• When STORAGE_URI is set, KServe's admission webhook injects the storage-initializer and creates the shared /mnt/models volume itself, as a plain (disk-backed) emptyDir with no Medium field.
• There is no configuration knob — neither per-InferenceService nor cluster-wide — to change that volume's medium. Verified against the KServe admin-guide config (the storageInitializer ConfigMap section exposes only image/cpu/memory/caBundle/modelcar keys, nothing for volume medium) and the injector source (pkg/webhook/admission/pod/storage_initializer_injector.go).
• Adding our own medium: Memory mount at /mnt/models collides with the injected one — pod is rejected with spec.containers[0].volumeMounts[...].mountPath: "/mnt/models": must be unique.

To use a ramdisk we would have to bypass the storage-initializer: drop
STORAGE_URI, define our own memory-backed emptyDir at /mnt/models, and add a
custom init container that downloads from S3 and replicates the S3 credential
wiring KServe normally injects.
Caveats that make this unattractive:

• A tmpfs emptyDir is charged against the pod's memory cgroup, so the container memory limit must be raised above the full model size (current limit is 32Gi; weights are ~27GB) or we just trade disk-pressure evictions for OOM kills.
• Limited upside: vLLM reads the weights off /mnt/models once at startup and loads them into GPU VRAM; serving never touches the path again. tmpfs only saves a one-time disk read while permanently pinning ~27GB of RAM.

Option B (recommended direction): stream S3 -> VRAM, no storage at all

vLLM can load weights directly from S3 into GPU memory using the Run:ai Model
Streamer, with no local disk staging:
https://docs.vllm.ai/en/stable/models/extensions/runai_model_streamer/

• --load-format runai_streamer (or runai_streamer_sharded) with the model set to an s3:// URI. Path is S3 -> bounded pinned CPU buffer -> GPU VRAM.
• No storage-initializer, no /mnt/models download, no ramdisk needed — this removes the disk-pressure root cause entirely rather than working around it.
• The CPU buffer size/concurrency is capped via --model-loader-extra-config '{"memory_limit": <bytes>, "concurrency": 16}', so it fits comfortably under the pod memory limit (it is not the full model).

Prerequisites / open questions before we can adopt it:

• The inference image needs the runai-model-streamer package (vllm[runai]) and the Liftwing entrypoint/wrapper must pass --load-format runai_streamer with the s3:// model arg. Our current spec drives everything via env vars (STORAGE_URI, MODEL_PATH=/mnt/models), which implies a wrapper that assumes a local model dir — so this is likely a change in the inference-services image, not just deployment-charts values.
• Weights must be safetensors (Run:ai Streamer requirement) — Qwen FP8 should qualify, to be confirmed.
• wmf-ml-models is a non-AWS S3-compatible endpoint, so it needs: AWS_ENDPOINT_URL, RUNAI_STREAMER_S3_USE_VIRTUAL_ADDRESSING=0, AWS_EC2_METADATA_DISABLED=true, plus the existing S3 access key/secret.
• Runs on MI300x/ROCm — the streamer read path is vendor-agnostic and should work, but must be verified on a pod since published examples are CUDA.

Recommendation / ranking

1. Streaming into VRAM (Option B) — best long-term fix; eliminates the disk dependency and scales to larger models. Needs image-side support, so it's a "v2" fix rather than immediate.
2. FS extension — quickest unblock for now, but temporary (see headroom note).
3. tmpfs bypass (Option A) — most complex, least upside; not recommended.

In the meantime vg0/kubelet was resized with lvextend --resizefs -l +100%FREE /dev/vg0/kubelet on ml-serve101[2-5]
which should alleviate the immediate disk pressure issue.
I will look into persisting these changes.

Change #1307368 had a related patch set uploaded (by Dpogorzelski; author: Dpogorzelski):

[operations/puppet@production] ml-serve: bigger kubelet LV for ml-serve

https://gerrit.wikimedia.org/r/1307368

Change #1307707 had a related patch set uploaded (by Bartosz Wójtowicz; author: Bartosz Wójtowicz):

[operations/deployment-charts@master] ml-services: Declare ephemeral-storage for llm-qwen3-14b, move to ml-serve1013

https://gerrit.wikimedia.org/r/1307707

Change #1307707 merged by jenkins-bot:

[operations/deployment-charts@master] ml-services: Declare ephemeral-storage for llm-qwen3-14b, move to ml-serve1013

https://gerrit.wikimedia.org/r/1307707

Change #1307723 had a related patch set uploaded (by Bartosz Wójtowicz; author: Bartosz Wójtowicz):

[operations/deployment-charts@master] admin_ng: raise llm namespace compute quota on ml-serve

https://gerrit.wikimedia.org/r/1307723

Change #1307732 had a related patch set uploaded (by Bartosz Wójtowicz; author: Bartosz Wójtowicz):

[operations/deployment-charts@master] ml-services: declare ephemeral-storage for qwen3-14b, move to ml-serve1013

https://gerrit.wikimedia.org/r/1307732

Change #1307723 merged by jenkins-bot:

[operations/deployment-charts@master] admin_ng: raise llm namespace compute quota on ml-serve

https://gerrit.wikimedia.org/r/1307723

Change #1307732 merged by jenkins-bot:

[operations/deployment-charts@master] ml-services: declare ephemeral-storage for qwen3-14b, move to ml-serve1013

https://gerrit.wikimedia.org/r/1307732

Change #1307368 merged by Dpogorzelski:

[operations/puppet@production] ml-serve: enlarge kubelet LV on GPU nodes

https://gerrit.wikimedia.org/r/1307368

Change #1319968 had a related patch set uploaded (by Dpogorzelski; author: Dpogorzelski):

[operations/puppet@production] ml-serve: lower kubelet LV target to 495G

https://gerrit.wikimedia.org/r/1319968

Change #1319968 merged by Dpogorzelski:

[operations/puppet@production] ml-serve: lower kubelet LV target to 495G

https://gerrit.wikimedia.org/r/1319968

FYI we have added ephemeral-storage requests/limits to all predictor containers that we are serving on the LiftWing cluster and everything is now redeployed with storage declared on staging and prod clusters. It was tracked here: https://phabricator.wikimedia.org/T431089.

There are some sidecars that still don't declare the storage, but the effect should be very minimal compared to predictors (queue-proxy at ~28Ki and istio-proxy at ~7MB). We're also planning to tackle those + enforce the ephemeral-storage to be declared on the chart level in a follow-up (tracked here), but we currently do not treat this as high priority.