This is extremely weird and should be looked into, from the alert:
https://alerts.wikimedia.org/?q=team%3Dwmcs&q=alertname%3DJobsEmailerNoEmails
https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/JobsEmailerNoEmails
It might be that in toolsbeta the job that was generating emails got stopped somehow.
@fgiunchedi on irc, probably not related but just in case:
re: T432115 JFYI earlier this morning I deployed dumps-nfs lb to toolsbeta and did a worker roll-reboot + a bunch of reboots for standalone hosts, not sure tbh how/if that would affect the mailer
The job that generates emails in toolsbeta seems stuck:
toolsbeta.sample-complex-app@toolsbeta-bastion-7:~$ toolforge jobs show generate-emails +---------------+-----------------------------------------------------------------+ | Job name: | generate-emails | +---------------+-----------------------------------------------------------------+ | Command: | test | +---------------+-----------------------------------------------------------------+ | Job type: | scheduled: 00,30 * * * * | +---------------+-----------------------------------------------------------------+ | Image: | python3.13 | +---------------+-----------------------------------------------------------------+ | Port: | N/A | +---------------+-----------------------------------------------------------------+ | File log: | no | +---------------+-----------------------------------------------------------------+ | Output log: | | +---------------+-----------------------------------------------------------------+ | Error log: | | +---------------+-----------------------------------------------------------------+ | Emails: | all | +---------------+-----------------------------------------------------------------+ | Resources: | default | +---------------+-----------------------------------------------------------------+ | Replicas: | N/A | +---------------+-----------------------------------------------------------------+ | Mounts: | all | +---------------+-----------------------------------------------------------------+ | Retry: | no | +---------------+-----------------------------------------------------------------+ | Timeout: | no | +---------------+-----------------------------------------------------------------+ | Health check: | N/A | +---------------+-----------------------------------------------------------------+ | Status: | Running for 5h46m17s | +---------------+-----------------------------------------------------------------+ | Hints: | Last run at 2026-07-14T09:00:00Z. Pod in 'Pending' phase. State | | | 'waiting'. Reason 'ContainerCreating'. | +---------------+-----------------------------------------------------------------+
The cronjob too:
toolsbeta.sample-complex-app@toolsbeta-bastion-7:~$ toolforge jobs list +-----------------+--------------------------+---------------------+ | Job name: | Job type: | Status: | +-----------------+--------------------------+---------------------+ | api | continuous | Running | | backend | continuous | Running | | backend-api | continuous | Running | | backend-worker | continuous | Running | | cronjob | scheduled: * * * * * | Running for 6h10m3s | | generate-emails | scheduled: 00,30 * * * * | Running for 5h45m3s | +-----------------+--------------------------+---------------------+
Yep, something failing with mounts:
apiVersion: v1 │ │ count: 180 │ │ eventTime: null │ │ firstTimestamp: "2026-07-14T09:00:00Z" │ │ involvedObject: │ │ apiVersion: v1 │ │ kind: Pod │ │ name: generate-emails-29733660-rhlgw │ │ namespace: tool-sample-complex-app │ │ resourceVersion: "1092741590" │ │ uid: 20140538-f5a5-4fee-a337-568e88300133 │ │ kind: Event │ │ lastTimestamp: "2026-07-14T14:50:00Z" │ │ message: 'MountVolume.SetUp failed for volume "dumpsrc-clouddumps1002" : hostPath │ │ type check failed: /mnt/nfs/dumps-clouddumps1002.wikimedia.org is not a directory' │ │ metadata: │ │ creationTimestamp: "2026-07-14T09:00:00Z" │ │ name: generate-emails-29733660-rhlgw.18c21c5fe540bb6e │ │ namespace: tool-sample-complex-app │ │ resourceVersion: "1092898570" │ │ uid: ba05e3dd-be2a-481a-8d83-21d610e514a0 │ │ reason: FailedMount │ │ reportingComponent: kubelet │ │ reportingInstance: toolsbeta-test-k8s-worker-nfs-10 │ │ source: │ │ component: kubelet │ │ host: toolsbeta-test-k8s-worker-nfs-10 │ │ type: Warning
This is sorted out, I'll create a task for the pods getting stuck in waiting for 5h+ and see how we surface that issue.