Same issue today (3 June). Both job-g131week and job-g13-elig were skipped again.

And both reports were skipped again tonight for the 4th day in a row. We work with a 7 day lag between when the G13 soon report is issued and when action is required and we're now down to a three day supply of reports until we are really in trouble.

This seems like a re-occurance of T308300: toolforge-jobs: Kubernetes scheduler overloaded at 00:00 each day causing missed jobs. The tl;dr of the problem is that Kubernetes can't start all of the jobs scheduled for midnight at exactly midnight since the scheduler can't take them all at once, and at some point it thinks it's been too long from the original scheduled time and skips that execution. I've added a couple of subtasks that we can do on the infrastructure side to avoid this from happening, but the best fix at the moment is to not schedule jobs at exactly midnight or top of the hour when the scheduler is less busy.

So I've seen what I suspect is the same issue, the potd tool's "send" job was never triggered on 2024-01-06 at 2:00; and the tfaprotbot tool's "tfasemibot" job was never triggered on 2023-12-28 at 23:00. If the problem is running on the hour, how much should I offset by? Is it likely to into issues if I have it run at, e.g. 22:59? I note that T338134: Use a higher `startingDeadlineSeconds` for less frequent jobs is still open - is that worth pursuing still? Should there be a subtask about non-scheduled jobs not sending any notifications as well?

The tfasemibot job is idempotent so I'm could just up the frequency, but that seems like it would make the overall problem worse by needlessly adding load to the cluster, no?

In T338006#9448777, @Legoktm wrote:

So I've seen what I suspect is the same issue, the potd tool's "send" job was never triggered on 2024-01-06 at 2:00; and the tfaprotbot tool's "tfasemibot" job was never triggered on 2023-12-28 at 23:00. If the problem is running on the hour, how much should I offset by? Is it likely to into issues if I have it run at, e.g. 22:59? I note that T338134: Use a higher `startingDeadlineSeconds` for less frequent jobs is still open - is that worth pursuing still?

Yes, I think that's still a good idea. The cron parsing added for T331684: Provide a means to introduce "skew" for scheduled jobs to avoid thundering herd problems should make it easier to implement.

Should there be a subtask about non-scheduled jobs not sending any notifications as well?

Not sure - jobs-emailer is in need of some love but even then sending notifications about things that did not happen is going to be quite difficult.