Page MenuHomePhabricator

Missing some email notifications from Phabricator (2024-07-17)
Closed, InvalidPublic

Description

I am missing some email notifications from Phabricator today.

For example, in this task: https://phabricator.wikimedia.org/T191156

I got emails about these comments:

But I did not get emails about these:

Event Timeline

matmarex triaged this task as Unbreak Now! priority.EditedJul 17 2024, 7:00 PM

I rely on the notifications for my work. I hope the missing ones will be found wherever they got lost in the pipes and resent. It is impossible to tell what I missed in tasks where I didn't get any notifications at all.

To rule out some stuff:

  • Any smaller upstream changes I made to Mailer code were after the upstream code version that we deployed on 2024-07-16 in T370109
  • https://phabricator.wikimedia.org/daemon/ says that PhabricatorMetaMTAWorker has been running recently; Leased Tasks show no Failures.
  • I received email notifications for all six comments listed in the task description (I'm in UTC+2), so I assume stuff seems to be generally working. Last line in those email notifications lists the task subscribers as Cc: Novem_Linguae, Dogu, matmarex, ... in all cases as expected.

Screenshot from 2024-07-17 21-49-53.png (250×1 px, 72 KB)

I can see the missing notifications at https://phabricator.wikimedia.org/notification/, by the way.

They don't seem to be in spam or trash in my email inbox. I don't think I have any filters that would delete them completely.

I wonder if you have any low-level logs that could confirm which emails were sent to me?

I'll also ask my email provider support, maybe it's something on their side.

matmarex lowered the priority of this task from Unbreak Now! to Needs Triage.Jul 17 2024, 8:22 PM

Huh, some of the missing messages arrived just now. They have Received: headers which, if I'm reading them right, seem to say they were indeed stuck somewhere for some hours (I can share a copy if you;re curious). The problem might be resolved.

Sorry about the alarm. Looks like it wasn't your fault: https://fastmailstatus.com/clypu668y362151han131th0gqi

Nah, better to be safe here. Glad you found the reason!

We spent some time digging around in the database and learned a few things about how this stuff is stored, at any rate.

Just for what it's worth:

select count(*) from metamta_mail where relatedPHID = 'PHID-TASK-2iacqyb64oiku3uw35ct' and parameters like '%@matmarex%';
+----------+
| count(*) |
+----------+
|     2736 |
+----------+
1 row in set (0.555 sec)

select count(*) from metamta_mail where relatedPHID = 'PHID-TASK-2iacqyb64oiku3uw35ct';
+----------+
| count(*) |
+----------+
|     2736 |
+----------+
1 row in set (0.002 sec)

(parameters is a JSON blob with a bunch of stuff including message recipients.)

So yeah, couldn't find any evidence of anything on this end. Relieved to see it was elsewhere.

I'm also a Fastmail customer, and have been noticing some delivery issues lately. May be a bit of a pattern there.

For future reference: https://wikitech.wikimedia.org/wiki/Phabricator/Mail_debugging

(Could use some work, but maybe it's useful breadcrumbs for someone in future.)