Page MenuHomePhabricator

Investigate why a coordinator has not received some "Wikipedia Library applications await your review" emails
Closed, DuplicatePublic

Description

We have a comment in our project talk page that asks about coordinator emails.

I review applications for some library resources. Usually I get an email every week or so. Today I got an email that said I had twelve applications and some of them were dating back to the 6th of October. I'm not sure how often it's supposed to drop me an email but this seems like maybe not often enough? I did check spam/trash, I hadn't gotten another email from Wikipedia since September.

We should investigate if the emails are sent once a month, or if they are sent more often, investigate why they are not being sent in a timely fashion.

Event Timeline

As a first step. I verified that the cron jobs are getting fired:

image.png (348×1 px, 81 KB)

I verified that we generated the messages for @Jessamyn on 10/13 and 10/20, which is the reported gap:

Wikipedia Library applications await your review	Wikipedia Library Card Platform <noreply@wikipedialibrary.wmflabs.org>	Jessamyn	Sent	Oct. 20, 2023, 12:30 a.m.	Oct. 20, 2023, 12:30 a.m.
Wikipedia Library applications await your review	Wikipedia Library Card Platform <noreply@wikipedialibrary.wmflabs.org>	Jessamyn	Sent	Oct. 13, 2023, 12:25 a.m.	Oct. 13, 2023, 12:25 a.m.

That at least eliminates an outage for the wikipedia lbrary itself as a source of the problem. The next thing to look for in that time period would be an email delivery issue.

@Jessamyn please let us know if you don't receive a notification on 11/3.

jsn.sherman changed the task status from Open to Stalled.Nov 3 2023, 2:32 PM
jsn.sherman claimed this task.

I did some looking through announcements and phab tasks to see if I could find any reported maintenance or outage that might indicate that we didn't get the message to the smtp relay and came up empty. The closest thing I could find was a dns change for cloud vps instances in september, but if that were a problem, all mail delivery would have been impacted.

We did have a per-user-account-scope issue pop up in october, which is when unconfirmed email addresses were removed from oauth responses:
https://phabricator.wikimedia.org/T283456

It looks like we're checking to see if we have a non-empty email address before trying to send, so I don't think that's it either:
https://github.com/WikipediaLibrary/TWLight/blob/b4df82f92b9abe81d94b8d400954f192c19dd1e2/TWLight/applications/management/commands/send_coordinator_reminders.py#L77-L84

Email receipt depends on a whole lot of things going right across multiple organizations and networks; if it looks like it's working moving forward from today, I think we'll just file it under "unknown ephemeral issue outside of our control"

Marking as stalled until we hear back about that.

I definitely got the email today. When's the next one scheduled?

jsn.sherman moved this task from In Progress to Done on the Moderator-Tools-Team (Kanban) board.

I definitely got the email today. When's the next one scheduled?

We send them out weekly, so 11/10. I'm marking this as resolved, but please reopen it if the issue pops back up.

jsn.sherman renamed this task from Investigate why "Wikipedia Library applications await your review" emails are belated to Investigate why a coordinator has not received some "Wikipedia Library applications await your review" emails.Nov 3 2023, 3:41 PM

If this is a recurring issue I wonder if this is related to T347512.

I definitely did NOT get the email on November 10th. (also did not get BL access codes if that's helpful).

Well, it's definitely looking like a delivery issue then; the system created the message and queued it for delivery at 12:35am GMT on nov 10.

jessamyn_reminder.png (819×1 px, 100 KB)

It might be worth seeing if the issues impacting the beta cluster might also be impacting vps hosting? T343925#9264468