Page MenuHomePhabricator
Create Task
Maniphest T332571

Request increased quota for itnsyn Toolforge tool
Closed, InvalidPublic
Actions

Description

Tool Name: itnsyn
Quota increase requested: +1 CPU.
Reason: Doesn't work after migration from Grid (T319823)

tools.itnsyn@tools-sgebastion-10:~$ kubectl get events                                                                                                       
LAST SEEN   TYPE      REASON             OBJECT                 MESSAGE
3m10s       Warning   FailedCreate       job/itn-syn-27988440   (combined from similar events): Error creating: pods "itn-syn-27988440-rn48h" is forbidden: exceeded quota: tool-itnsyn, requested: limits.cpu=500m, used: limits.cpu=2, limited: limits.cpu=2
26m         Normal    JobAlreadyActive   cronjob/itn-syn        Not starting job because prior execution is running and concurrency policy is Forbid

also see:
https://grafana.wmcloud.org/d/TJuKfnt4z/kubernetes-namespace?orgId=1&var-namespace=tool-itnsyn

Related Objects
Search...

Event Timeline

Shizhao created this task.Mar 20 2023, 12:38 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 20 2023, 12:38 PM
Shizhao added a parent task: T319823: Migrate itnsyn from Toolforge GridEngine to Toolforge Kubernetes.Mar 20 2023, 12:39 PM
Shizhao mentioned this in T332477: Migrate Shizhao's tools from Toolforge GridEngine to Toolforge Kubernetes.Mar 20 2023, 12:41 PM
Kizule subscribed.Mar 20 2023, 2:17 PM
taavi moved this task from Inbox to Discussion needed on the Toolforge (Quota-requests) board.Mar 20 2023, 2:17 PM
taavi subscribed.

Hi! As far as I can tell you were running out of quota because you had several webservice shell instances open. After closing those it looks like your tool can launch those jobs just fine. Do you still think you would need more quota?

Shizhao added a comment.Mar 20 2023, 2:34 PM

In order for the service to run stably, I have temporarily rolled back 2 jobs to GridEngine

Shizhao added a comment.Mar 20 2023, 2:56 PM
在T332571#8710216中，@taavi写道：

Hi! As far as I can tell you were running out of quota because you had several webservice shell instances open. After closing those it looks like your tool can launch those jobs just fine. Do you still think you would need more quota?

I only have one webservice running. Why are there several webservice shell instances open? How can I closing them?

tools.itnsyn@tools-sgebastion-10:~$ webservice status
Your webservice of type php7.4 is running on backend kubernetes
Shizhao mentioned this in T319823: Migrate itnsyn from Toolforge GridEngine to Toolforge Kubernetes.Mar 21 2023, 3:17 AM
Shizhao added a comment.Mar 22 2023, 2:06 PM

ok. Use kubectl delete pod [my-pod] to kill unnecessary pods without exceeding the limit.

Shizhao closed this task as Invalid.Mar 23 2023, 2:44 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits