Page MenuHomePhabricator
Create Task
Maniphest T301287

Kuberbernetes cronjobs for william-avery-bot no longer being scheduled on Toolforge
Closed, ResolvedPublic
Actions

Description

The kubernetes cronjob that runs my bot on toolforge has not been scheduled to run since about 2022-02-07 14:00

It had previously been running three times an hour for the previous 76 days or so.

I am able to start a shell with the webservice command and ran the python command successfully without a problem.

Output from kubectl is as follows:

tools.william-avery-bot@tools-sgebastion-07:~$ kubectl get cronjobs
NAME SCHEDULE SUSPEND ACTIVE LAST SCHEDULE AGE
taxonbardesyncer 9,29,49 * * * * False 0 29h 77d

Related Objects

Event Timeline

William_Avery created this task.Feb 8 2022, 8:44 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 8 2022, 8:44 PM
taavi subscribed.Feb 8 2022, 8:53 PM

Your tool has a large amount of completed Job objects that have not been cleaned up:

taavi@tools-sgebastion-10:~ $ kubectl get job -n tool-william-avery-bot 
NAME                           COMPLETIONS   DURATION   AGE
recombination-fix              1/1           142m       86d
recombination-ocs-q147877      1/1           11m        83d
recombination-ocs-q2068481     1/1           45s        82d
recombination-ocs-q22671       1/1           41m        79d
recombination-ocs-q25364       1/1           8m59s      79d
recombination-ocs-q26371       1/1           13m        79d
recombination-ocs-q28319       1/1           6m58s      81d
recombination-ocs-q7377        1/1           20m        80d
recombination-scan-q128257     1/1           3m9s       85d
recombination-scan-q147877     1/1           171m       85d
recombination-scan-q18710482   1/1           2m23s      85d
recombination-scan-q2068481    1/1           22m        84d
recombination-scan-q22651      1/1           44m        78d
recombination-scan-q22671      1/1           85m        79d
recombination-scan-q25364      1/1           31m        80d
recombination-scan-q26371      1/1           27m        79d
recombination-scan-q4982434    1/1           30m        85d
recombination-scan-q5113       1/1           105m       85d
recombination-scan-q60500115   1/1           24s        85d
recombination-scan-q655316     1/1           49m        84d
recombination-scan-q6653350    1/1           48s        85d
recombination-scan-q7377       1/1           66m        81d
recombination-scan-q756153     1/1           77m        84d
recombination-scan-q758663     1/1           10s        85d
recombination-scan-q870378     1/1           10s        84d
recombination-scan-q972302     1/1           36m        84d
redirectclassifier             1/1           13m        51d
taxonbardesyncer-1644206940    0/1           40h        40h
taxonbardesyncer-1644241740    1/1           19s        30h
taxonbardesyncer-1644242940    1/1           18s        30h
taxonbardesyncer-1644244140    1/1           20s        30h

We recently fixed a typo in the default per-tool quotas after a single tool caused major issues when it spawned thousands of Job objects (T301081). This is what's preventing your tool from launching new job objects:

taavi@tools-sgebastion-10:~ $ kubectl describe quota -n tool-william-avery-bot
Name:                   tool-william-avery-bot
Namespace:              tool-william-avery-bot
Resource                Used  Hard
--------                ----  ----
count/cronjobs.batch    1     50
count/jobs.batch        31    15
[non-relevant items not included]

My suggested fix would be to configure a lower ttlSecondsAsFinished value on the jobs as documented here: https://kubernetes.io/docs/concepts/workloads/controllers/job/#ttl-mechanism-for-finished-jobs (documented as a Kubernetes 1.21+ feature, but it has been enabled on our cluster that's currently on 1.20.)

William_Avery added a comment.Feb 8 2022, 9:09 PM

Thank you so much for this prompt and informative reply.

I have deleted all those old "recombination" jobs and the bot cronjob running again now.

I will look into the ttlSecondsAsFinished before I run any further batch jobs.

William_Avery closed this task as Resolved.Feb 8 2022, 9:09 PM
William_Avery claimed this task.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits