Page MenuHomePhabricator
Create Task
Maniphest T426947

Raise revertrisk namespace resource quota to allow rolling deployments
Closed, ResolvedPublic
Actions

Description

Description:

Rollouts in the revertrisk namespace stall on exceeded quota: the new Knative revision comes up before the old one goes away, and their combined requests don't fit. Pods sit Pending and the namespace is left mixed-version until the old revision is deleted by hand.

Steady-state usage is ~98 CPU / ~148Gi against a 140 CPU / 200Gi quota — not enough headroom for one service to double up during a rollout. Raised the revertrisk quota to 160 CPU / 230Gi in CL 1326234 (ml-serve-eqiad, ml-serve-codfw, ml-staging-codfw).

Scoped to revertrisk. Quotas in the other ml-serve namespaces and GPU rollout headroom (T420883) are not covered here.

Details

Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
admin_ng: raise revertrisk namespace resource quota for rollout headroomAikoChouoperations/deployment-chartsmaster+4 -4
Customize query in gerrit

Related Objects

Event Timeline

achou created this task.May 21 2026, 10:51 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 21 2026, 10:51 AM
achou added a comment.Jul 31 2026, 2:52 PM

While testing the rollout in T420883, I found another deployment constraint that's related to this discussion.

Rolling updates for GPU-backed Knative services temporarily require additional GPU capacity because the old and new revisions coexist during the rollout. In our case, this prevented the rollout from completing even though the steady state would fit within the available GPUs.

I've documented the details in T420883#12174854, but this suggests we should also consider GPU headroom alongside CPU/memory quota when planning rolling updates.

achou moved this task from Unsorted/Needs Triage to Ready to Go on the Machine-Learning-Team (Q1 FY2026-27) board.Aug 4 2026, 1:10 PM
isarantopoulos added a project: Essential-Work.Aug 4 2026, 1:10 PM
gerritbot added a comment.Mon, Aug 17, 9:54 AM

Change #1326234 had a related patch set uploaded (by AikoChou; author: AikoChou):

[operations/deployment-charts@master] admin_ng: raise revertrisk namespace resource quota for rollout headroom

https://gerrit.wikimedia.org/r/1326234

gerritbot added a project: Patch-For-Review.Mon, Aug 17, 9:54 AM
achou moved this task from Ready to Go to In Progress on the Machine-Learning-Team (Q1 FY2026-27) board.Mon, Aug 17, 9:56 AM
gerritbot added a comment.Mon, Aug 17, 3:34 PM

Change #1326234 merged by jenkins-bot:

[operations/deployment-charts@master] admin_ng: raise revertrisk namespace resource quota for rollout headroom

https://gerrit.wikimedia.org/r/1326234

Maintenance_bot removed a project: Patch-For-Review.Mon, Aug 17, 4:31 PM
achou renamed this task from ml-serve: assess and raise resource quotas to support rolling deployments in revertrisk (and other) namespaces to Raise revertrisk namespace resource quota to allow rolling deployments.Wed, Aug 19, 10:27 AM
achou closed this task as Resolved.
achou claimed this task.
achou updated the task description. (Show Details)

Deployed 2026-08-17, verified with kubectl describe quota quota-compute-resources. Resolving.

achou moved this task from In Progress to Done on the Machine-Learning-Team (Q1 FY2026-27) board.Wed, Aug 19, 10:29 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits