Today, @Michael discovered there is a long-running mw-cron job for the Growth team:
migr@deploy2002:~$ kubectl logs -f job/growthexperiments-listtaskcounts-29513771 mediawiki-main-app Error from server (BadRequest): container "mediawiki-main-app" in pod "growthexperiments-listtaskcounts-29513771-jrv9k" is waiting to start: trying and failing to pull image migr@deploy2002:~$ kubectl get jobs -l team=growth,cronjob=growthexperiments-listtaskcounts NAME STATUS COMPLETIONS DURATION AGE growthexperiments-listtaskcounts-29513771 Running 0/1 33d 33d migr@deploy2002:~$
This job should never take more than a couple of minutes to complete. If it takes more than an hour, we definitely have a problem.
Would it be possible to add something like max_expected_length = XX hours to the job definition, and trigger an alert if the expected duration was exceeded? This would help us notice this much quickly than manually after a month.
Update 2026-03-19 - One alternative to passive alerting that we may want to consider is functionality for setting activeDeadlineSeconds on the cronjob's job template. That would provide an active backstop on execution time. As a side effect, the job failure alert should also fire, as a result of the job being terminated. See T420354#11722534.