Page MenuHomePhabricator
Create Task
Maniphest T401833

Cannot deploy function-orchestrator in staging environment due to insufficient quotas
Closed, ResolvedPublic
Actions

Description

The name of the appArmor field has apparently changed. Error log:

ERROR:
  exit status 1

EXIT STATUS
  1

STDERR:
  W0813 14:10:13.216701 2851566 warnings.go:70] spec.template.metadata.annotations[container.apparmor.security.beta.kubernetes.io/function-orchestrator-main-orchestrator]: deprecated since v1.30; use the "appArmorProfile" field instead
  W0813 14:20:14.368859 2851566 warnings.go:70] spec.template.metadata.annotations[container.apparmor.security.beta.kubernetes.io/function-orchestrator-main-orchestrator]: deprecated since v1.30; use the "appArmorProfile" field instead
  Error: UPGRADE FAILED: release main-orchestrator failed, and has been rolled back due to atomic being set: context deadline exceeded

COMBINED OUTPUT:
  W0813 14:10:13.216701 2851566 warnings.go:70] spec.template.metadata.annotations[container.apparmor.security.beta.kubernetes.io/function-orchestrator-main-orchestrator]: deprecated since v1.30; use the "appArmorProfile" field instead
  W0813 14:20:14.368859 2851566 warnings.go:70] spec.template.metadata.annotations[container.apparmor.security.beta.kubernetes.io/function-orchestrator-main-orchestrator]: deprecated since v1.30; use the "appArmorProfile" field instead
  Error: UPGRADE FAILED: release main-orchestrator failed, and has been rolled back due to atomic being set: context deadline exceeded

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
wikifunctions: Bump staging quota to 20Goperations/deployment-chartsmaster+4 -4
admin: Brown paper bag fix for wikifunctionsoperations/deployment-chartsmaster+4 -4
wikifunctions: Pull down the memory limits for staging instancesoperations/deployment-chartsmaster+6 -0
staging: Bump wikifunctions quotasoperations/deployment-chartsmaster+45 -1
Customize query in gerrit

Event Timeline

cmassaro created this task.Aug 13 2025, 2:25 PM
cmassaro updated the task description. (Show Details)
akosiaris subscribed.Aug 13 2025, 2:49 PM

Those are warning (note the W prefix). They wouldn't stop the deployment from happening.

The actual reason is this https://logstash.wikimedia.org/app/discover#/doc/logstash-*/logstash-k8s-1-7.0.0-1-2025.08.13?id=8Q7Uo5gBgiE0yhV9mhEm

Pasting for convenience

(combined from similar events): Error creating: pods "function-orchestrator-main-orchestrator-df5fdb7c9-dmrg5" is forbidden: exceeded quota: quota-compute-resources, requested: limits.memory=4172Mi, used: limits.memory=7220Mi, limited: limits.memory=10Gi

Simply put, the namespace in staging isn't provisioned for pods that are this large. Resources in staging are scarce. We can probably bump the quotas up a bit, but probably just enough to allow this to run.

akosiaris renamed this task from Cannot deploy function-orchestrator due to deprecated appArmor field to Cannot deploy function-orchestrator in staging environment due to insufficient quotas.Aug 13 2025, 2:54 PM
gerritbot added a comment.Aug 13 2025, 3:00 PM

Change #1178566 had a related patch set uploaded (by Alexandros Kosiaris; author: Alexandros Kosiaris):

[operations/deployment-charts@master] staging: Bump wikifunctions quotas

https://gerrit.wikimedia.org/r/1178566

gerritbot added a project: Patch-For-Review.Aug 13 2025, 3:00 PM
Jdforrester-WMF subscribed.Aug 13 2025, 3:15 PM
In T401833#11082897, @akosiaris wrote:

Those are warning (note the W prefix). They wouldn't stop the deployment from happening.

The actual reason is this https://logstash.wikimedia.org/app/discover#/doc/logstash-*/logstash-k8s-1-7.0.0-1-2025.08.13?id=8Q7Uo5gBgiE0yhV9mhEm

Pasting for convenience

(combined from similar events): Error creating: pods "function-orchestrator-main-orchestrator-df5fdb7c9-dmrg5" is forbidden: exceeded quota: quota-compute-resources, requested: limits.memory=4172Mi, used: limits.memory=7220Mi, limited: limits.memory=10Gi

Simply put, the namespace in staging isn't provisioned for pods that are this large. Resources in staging are scarce. We can probably bump the quotas up a bit, but probably just enough to allow this to run.

Aha, we can pull the memory for staging back down. I suppose the issue is the switch-over with the new pods and the old ones, which is why the change to increase the memory itself deployed fine, but the next deploy failed?

gerritbot added a comment.Aug 13 2025, 3:17 PM

Change #1178566 merged by jenkins-bot:

[operations/deployment-charts@master] staging: Bump wikifunctions quotas

https://gerrit.wikimedia.org/r/1178566

gerritbot added a comment.Aug 13 2025, 3:20 PM

Change #1178571 had a related patch set uploaded (by Jforrester; author: Jforrester):

[operations/deployment-charts@master] wikifunctions: Pull down the memory limits for staging instances

https://gerrit.wikimedia.org/r/1178571

akosiaris added a comment.Aug 13 2025, 3:30 PM
In T401833#11082991, @Jdforrester-WMF wrote:
In T401833#11082897, @akosiaris wrote:

Those are warning (note the W prefix). They wouldn't stop the deployment from happening.

The actual reason is this https://logstash.wikimedia.org/app/discover#/doc/logstash-*/logstash-k8s-1-7.0.0-1-2025.08.13?id=8Q7Uo5gBgiE0yhV9mhEm

Pasting for convenience

(combined from similar events): Error creating: pods "function-orchestrator-main-orchestrator-df5fdb7c9-dmrg5" is forbidden: exceeded quota: quota-compute-resources, requested: limits.memory=4172Mi, used: limits.memory=7220Mi, limited: limits.memory=10Gi

Simply put, the namespace in staging isn't provisioned for pods that are this large. Resources in staging are scarce. We can probably bump the quotas up a bit, but probably just enough to allow this to run.

Aha, we can pull the memory for staging back down.

That's probably not needed. I bumped the quotas from 10G to 20G which should provide enough space for now.

I suppose the issue is the switch-over with the new pods and the old ones, which is why the change to increase the memory itself deployed fine, but the next deploy failed?

Yes. The deployment will start first and make sure new pods are ready and then kill new ones. Which in this cases means double the usage (cause it's 1+1 pod). Usually this is set to 25% (except MediaWiki which is on the very low single digits due to sheer size).

akosiaris added a comment.Aug 13 2025, 3:30 PM

Patches deployed, you should be good to retry @cmassaro

cmassaro added a comment.Aug 13 2025, 4:44 PM
In T401833#11083055, @akosiaris wrote:

Patches deployed, you should be good to retry @cmassaro

Thank you!

gerritbot added a comment.Aug 13 2025, 4:54 PM

Change #1178571 merged by jenkins-bot:

[operations/deployment-charts@master] wikifunctions: Pull down the memory limits for staging instances

https://gerrit.wikimedia.org/r/1178571

Jdforrester-WMF added a comment.Aug 13 2025, 5:07 PM

Unfortunately the new 20GiB limit doesn't seem to have taken: https://logstash.wikimedia.org/app/dashboards#/doc/logstash-*/logstash-k8s-1-7.0.0-1-2025.08.13?id=0JJhpJgBSq9zfLI0u-Q1 — is there something more we should do?

Maintenance_bot removed a project: Patch-For-Review.Aug 13 2025, 5:30 PM
gerritbot added a comment.Aug 14 2025, 8:57 AM

Change #1178817 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/deployment-charts@master] wikifunctions: Bump staging quota to 20G

https://gerrit.wikimedia.org/r/1178817

gerritbot added a project: Patch-For-Review.Aug 14 2025, 8:57 AM
gerritbot added a comment.Aug 14 2025, 9:35 AM

Change #1178821 had a related patch set uploaded (by Alexandros Kosiaris; author: Alexandros Kosiaris):

[operations/deployment-charts@master] admin: Brown paper bag fix for wikifunctions

https://gerrit.wikimedia.org/r/1178821

gerritbot added a comment.Aug 14 2025, 9:35 AM

Change #1178821 abandoned by Clément Goubert:

[operations/deployment-charts@master] admin: Brown paper bag fix for wikifunctions

Reason:

Done in https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1178817

https://gerrit.wikimedia.org/r/1178821

gerritbot added a comment.Aug 14 2025, 9:37 AM

Change #1178817 merged by jenkins-bot:

[operations/deployment-charts@master] wikifunctions: Bump staging quota to 20G

https://gerrit.wikimedia.org/r/1178817

Clement_Goubert subscribed.Aug 14 2025, 9:43 AM
In T401833#11083397, @Jdforrester-WMF wrote:

Unfortunately the new 20GiB limit doesn't seem to have taken: https://logstash.wikimedia.org/app/dashboards#/doc/logstash-*/logstash-k8s-1-7.0.0-1-2025.08.13?id=0JJhpJgBSq9zfLI0u-Q1 — is there something more we should do?

Should be fixed now.

Maintenance_bot removed a project: Patch-For-Review.Aug 14 2025, 10:30 AM
akosiaris added a comment.Aug 14 2025, 11:22 AM

Thanks @claime. Stupid typo on my side.

Jdforrester-WMF closed this task as Resolved.Aug 18 2025, 11:50 AM
Jdforrester-WMF assigned this task to akosiaris.
Jdforrester-WMF edited projects, added Essential-Work, Abstract Wikipedia team (26Q1 (Jul–Sep)); removed Abstract Wikipedia team.

Confirm this is now fixed via a deploy, thank you!

Jdforrester-WMF moved this task from Incoming to Needs Sign-off on the Abstract Wikipedia team (26Q1 (Jul–Sep)) board.Aug 18 2025, 11:50 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits