Change Details

Today, @Michael discovered there is a long-running mw-cron job for the Growth team: ``` migr@deploy2002:~$ kubectl logs -f job/growthexperiments-listtaskcounts-29513771 mediawiki-main-app Error from server (BadRequest): container "mediawiki-main-app" in pod "growthexperiments-listtaskcounts-29513771-jrv9k" is waiting to start: trying and failing to pull image migr@deploy2002:~$ kubectl get jobs -l team=growth,cronjob=growthexperiments-listtaskcounts NAME STATUS COMPLETIONS DURATION AGE growthexperiments-listtaskcounts-29513771 Running 0/1 33d 33d migr@deploy2002:~$ ``` This job should never take more than a couple of minutes to complete. If it takes more than an hour, we definitely have a problem. Would it be possible to add something like `max_expected_length = XX hours` to the job definition, and trigger an alert if the expected duration was exceeded? This would help us notice this much quickly than manually after a month.

Today, @Michael discovered there is a long-running mw-cron job for the Growth team: ``` migr@deploy2002:~$ kubectl logs -f job/growthexperiments-listtaskcounts-29513771 mediawiki-main-app Error from server (BadRequest): container "mediawiki-main-app" in pod "growthexperiments-listtaskcounts-29513771-jrv9k" is waiting to start: trying and failing to pull image migr@deploy2002:~$ kubectl get jobs -l team=growth,cronjob=growthexperiments-listtaskcounts NAME STATUS COMPLETIONS DURATION AGE growthexperiments-listtaskcounts-29513771 Running 0/1 33d 33d migr@deploy2002:~$ ``` This job should never take more than a couple of minutes to complete. If it takes more than an hour, we definitely have a problem. Would it be possible to add something like `max_expected_length = XX hours` to the job definition, and trigger an alert if the expected duration was exceeded? This would help us notice this much quickly than manually after a month. --- //Update 2026-03-19// - One alternative to passive alerting that we may want to consider is functionality for setting `activeDeadlineSeconds` on the cronjob's job template. That would provide an //active// backstop on execution time. As a side effect, the job failure alert should also fire, as a result of the job being terminated. See T420354#11722534.

Today, @Michael discovered there is a long-running mw-cron job for the Growth team: ``` migr@deploy2002:~$ kubectl logs -f job/growthexperiments-listtaskcounts-29513771 mediawiki-main-app Error from server (BadRequest): container "mediawiki-main-app" in pod "growthexperiments-listtaskcounts-29513771-jrv9k" is waiting to start: trying and failing to pull image migr@deploy2002:~$ kubectl get jobs -l team=growth,cronjob=growthexperiments-listtaskcounts NAME STATUS COMPLETIONS DURATION AGE growthexperiments-listtaskcounts-29513771 Running 0/1 33d 33d migr@deploy2002:~$ ``` This job should never take more than a couple of minutes to complete. If it takes more than an hour, we definitely have a problem. Would it be possible to add something like `max_expected_length = XX hours` to the job definition, and trigger an alert if the expected duration was exceeded? This would help us notice this much quickly than manually after a month. --- //Update 2026-03-19// - One alternative to passive alerting that we may want to consider is functionality for setting `activeDeadlineSeconds` on the cronjob's job template. That would provide an //active// backstop on execution time. As a side effect, the job failure alert should also fire, as a result of the job being terminated. See T420354#11722534.