Page MenuHomePhabricator

No email received for latest train presync
Closed, ResolvedPublic

Description

Scap's train presync runs every Tuesday on early UTC hours to ensure all MW hosts have already synced the latest MW version branch prior to the beginning of the weekly train.

The systemd timer is configured to send the results to releng@lists.wikimedia.org but we didn't receive an email for the 2024-02-20 run for 1.42.0-wmf.19 (T354437), even though the job seemed to execute correctly:

someun@dep:~$ systemctl list-timers --all | grep -e "PASSED\|train"
NEXT                         LEFT          LAST                         PASSED               UNIT                                                ACTIVATES
Tue 2024-02-27 04:00:00 UTC  6 days left   Tue 2024-02-20 04:00:00 UTC  11h ago              train-presync.timer                                 train-presync.service

A cursory look through recent puppet changes didn't reveal anything obvious that could have affected the email sending.

Event Timeline

Ah, that's why I missed it in the archives. Systemd says "PASSED" but it passed with failure? Anyway, I guess we got the email.

Anything else worth investigating here?

Though the message shows up in the archives, it didn't arrive in our mailboxes.

The email finally arrived in my inbox today at 02:27 UTC +1. Maybe this was a one-time hiccup on the Google side. I'm gonna keep an eye on this, if we get the email normally next week I'll close the task

There was a issue with mailman, lego did their magic earlier today and cleared the blacklog, see the backlog on irc in #-tech which may explain the delay.

jnuche claimed this task.

@Peachey88 thanks for the update. More details of the incident at T358020. Closing this task