Change Details

**Description:** During a recent image tag bump in the `revertrisk` namespace, Kubernetes was unable to replace all old pods with the new ones. The rollout stalled because the rolling update strategy creates new pods before terminating old ones — so at peak, both old and new pods are running simultaneously and the combined CPU/memory requests exceed the current namespace or cluster-level resource quota. **Steps to reproduce:** 1. Deploy a new image tag to the `revertrisk` namespace (e.g. via `helmfile sync`) 2. Observe that only some pods are replaced; the rest remain on the old image 3. Check `kubectl describe resourcequota -n revertrisk` — quota is exhausted mid-rollout **Impact:** Partial rollouts leave the namespace in a mixed-version state, requiring manual intervention to complete the deployment. This affects all namespaces with high replica counts (e.g. `revertrisk`, which runs 5–15 replicas). **Proposed work:** - Audit current CPU/memory `ResourceQuota` and `LimitRange` settings across all ml-serve namespaces, with focus on high-replica ones (revertrisk, etc.) - Determine the headroom needed to accommodate a full rolling update (i.e. quota should cover maxReplicas + maxSurge worth of resources) - Raise the namespace-level and/or overall cluster quota limits accordingly - Consider documenting the quota sizing policy so future replica count changes account for this **References:** - Deployment that triggered this: [[ https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1290699 | CL 1290699 ]] (revertrisk image bump, 2026-05-21)

**Description:** Rollouts in the revertrisk namespace stall on exceeded quota: the new Knative revision comes up before the old one goes away, and their combined requests don't fit. Pods sit Pending and the namespace is left mixed-version until the old revision is deleted by hand. Steady-state usage is ~98 CPU / ~148Gi against a 140 CPU / 200Gi quota — not enough headroom for one service to double up during a rollout. Raised the revertrisk quota to 160 CPU / 230Gi in [[ https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1326234 | CL 1326234 ]] (ml-serve-eqiad, ml-serve-codfw, ml-staging-codfw). Scoped to revertrisk. Quotas in the other ml-serve namespaces and GPU rollout headroom (T420883) are not covered here.

**Description:** During a recent image tag bump in the `revertrisk` namespace, Kubernetes was unable to replace all old pods with the new ones. The rollout stalled because the rolling update strategy creates new pods before terminating old ones — so at peak, both old and new pods are running simultaneously and the combined CPU/memory requests exceed the current namespace or cluster-level resource quota. **Steps to reproduce:** 1. Deploy a new image tag to the `revertrisk` namespace (e.g. via `helmfile sync`) 2. Observe that only some pods are replaced; the rest remain on the old image 3. Check `kubectl describe resourcequota -n revertrisk` — quota is exhausted mid-rollout **Impact:** Partial rollouts leave the namespace in a mixed-version state, requiring manual intervention to complete the deployment. This affects all namespaces with high replica counts (e.g. `revertrisk`, which runs 5–15 replicas). **Proposed work:** - Audit current CPU/memory `ResourceQuota` and `LimitRange` settings across all ml-serve namespaces, with focus on high-replica ones (revertriskRollouts in the revertrisk namespace stall on exceeded quota: the new Knative revision comes up before the old one goes away, etc.) - Determine the headroom needed to accommodate a full rolling update (i.eand their combined requests don't fit. quota should cover maxReplicas + maxSurge worth of resources) - Raise the namespace-level and/or overall cluster quota limits accordingly - Consider documenting the quota sizing policy so future replica count changes account for thisPods sit Pending and the namespace is left mixed-version until the old revision is deleted by hand. **References:**Steady-state usage is ~98 CPU / ~148Gi against a 140 CPU / 200Gi quota — not enough headroom for one service to double up during a rollout. Raised the revertrisk quota to 160 CPU / 230Gi in [[ https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1326234 | CL 1326234 ]] (ml-serve-eqiad, ml-serve-codfw, ml-staging-codfw). - Deployment that triggered this: [[ https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1290699 | CL 1290699 ]] (revertrisk image bump,Scoped to revertrisk. 2026-05-21)Quotas in the other ml-serve namespaces and GPU rollout headroom (T420883) are not covered here.