We got an email about this job; we should figure out what’s wrong and what we need to do about it. (AFAIK, a task is supposed to be created automatically as well, but I can’t find it.)
Description
Description
Related Objects
Related Objects
Event Timeline
Comment Actions
lucaswerkmeister-wmde@deploy2002 ~ $ kube-env mw-cron codfw lucaswerkmeister-wmde@deploy2002 ~ $ kubectl get jobs --field-selector status.successful=0 [snip other jobs] wikidata-resubmit-changes-for-dispatch-29415459 Failed 0/1 103m 103m lucaswerkmeister-wmde@deploy2002 ~ $ kubectl logs jobs/wikidata-resubmit-changes-for-dispatch-29415459 mediawiki-main-app extensions/Wikibase/repo/maintenance/ResubmitChanges.php: Start run The service mesh is unavailable, which can lead to unexpected results. Therefore, the script will not be executed. If you are *very* sure your script will not need the service mesh at all, you can run it again with MESH_CHECK_SKIP=1
That sounds like a transient error that should resolve itself? But is it going to happen automatically?
lucaswerkmeister-wmde@deploy2002 ~ $ kubectl get cronjob wikidata-resubmit-changes-for-dispatch NAME SCHEDULE TIMEZONE SUSPEND ACTIVE LAST SCHEDULE AGE wikidata-resubmit-changes-for-dispatch 39 * * * * <none> False 0 44m 71d
That looks like an hourly cronjob to me, yet the age given above is 103m, i.e. over an hour, and it seemingly didn’t re-run yet? Or am I missing something?
Comment Actions
https://wikitech.wikimedia.org/wiki/Mw-cron_jobs#Manually_deleting_a_failed_Job says:
In order to reset alerting, the failed job(s) need to be manually deleted.
Is that also necessary to make it run periodically again?
Comment Actions
Oh, nevermind, it is running:
lucaswerkmeister-wmde@deploy2002 ~ $ kubectl get jobs -l cronjob=wikidata-resubmit-changes-for-dispatch NAME STATUS COMPLETIONS DURATION AGE wikidata-resubmit-changes-for-dispatch-29415399 Complete 1/1 6s 3h wikidata-resubmit-changes-for-dispatch-29415459 Failed 0/1 120m 120m wikidata-resubmit-changes-for-dispatch-29415519 Complete 1/1 6s 60m wikidata-resubmit-changes-for-dispatch-29415579 Complete 1/1 12s 28s
So I assume all that’s left to do is to delete the failed job to resolve the alert, as mentioned above.
Comment Actions
Mentioned in SAL (#wikimedia-operations) [2025-12-05T11:42:55Z] <lucaswerkmeister-wmde@deploy2002> kubectl delete job wikidata-resubmit-changes-for-dispatch-29415459 # T411862
Comment Actions
The alert seems to be gone, so resolving. Reopen if there are still lingering issues :)
Comment Actions
(Some more discussion about this type of issue is apparently available at T410764, and T390972: Restart CronJobs on failure of the service mesh seemingly attempts to fix it.)
Comment Actions
AFAIK, a task is supposed to be created automatically as well
FWIW, from what I can see, it looks like the configured 'team' for that mw-cron job [1] is currently only set to send an email [2] [3].
[1] https://gerrit.wikimedia.org/g/operations/puppet/+/63a8174ffd28bba965aa9fed92ef99fcfe5a2da8/modules/profile/manifests/mediawiki/maintenance/wikidata.pp#7
[2] https://gerrit.wikimedia.org/g/operations/puppet/+/63a8174ffd28bba965aa9fed92ef99fcfe5a2da8/modules/alertmanager/templates/alertmanager.yml.erb#270
[3] https://gerrit.wikimedia.org/g/operations/puppet/+/63a8174ffd28bba965aa9fed92ef99fcfe5a2da8/modules/alertmanager/templates/alertmanager.yml.erb#575