Page MenuHomePhabricator

Phab and Gerrit emails stopped at around 1900 UTC 6th June
Closed, ResolvedPublic

Description

[02:20:56] <MatmaRex> i stopped receiving any email from gerrit and phabricator a couple hours ago. is it just me?
[02:21:16] <MatmaRex> e.g. i have no emails for any of the comments on https://phabricator.wikimedia.org/T196588
[02:22:01] <MatmaRex> the last one i received at was 19:12 UTC

Event Timeline

Reedy created this task.Jun 7 2018, 1:34 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 7 2018, 1:34 AM
Reedy triaged this task as High priority.Jun 7 2018, 1:34 AM
Reedy added a subscriber: herron.Jun 7 2018, 1:43 AM

Mentioned in SAL (#wikimedia-operations) [2018-06-07T02:00:29Z] <paravoid> starting exim4 and reenabling puppet on mx1001, due to T196598

faidon closed this task as Resolved.Jun 7 2018, 2:11 AM
faidon claimed this task.
faidon added a subscriber: faidon.

The cause was the prep for T175361, in combination with a couple of unexpected misconfigurations/SPOFs, given it's been years since the switchover from mx1001->mx2001 has been tested.

I restarted exim above which fixed this. Phabricator even queued the emails and is sending them now, but unfortunately I don't think that will be the case with others :(

I'll resolve this and we can follow up on the other task for the rest of the investigation.

Vvjjkkii renamed this task from Phab and Gerrit emails stopped at around 1900 UTC 6th June to 9hbaaaaaaa.Jul 1 2018, 1:05 AM
Vvjjkkii reopened this task as Open.
Vvjjkkii removed faidon as the assignee of this task.
Vvjjkkii updated the task description. (Show Details)
Vvjjkkii removed a subscriber: Aklapper.
CommunityTechBot renamed this task from 9hbaaaaaaa to Phab and Gerrit emails stopped at around 1900 UTC 6th June.Jul 2 2018, 8:58 AM
CommunityTechBot closed this task as Resolved.
CommunityTechBot assigned this task to faidon.
CommunityTechBot updated the task description. (Show Details)
CommunityTechBot added a subscriber: Aklapper.