Page MenuHomePhabricator

Gerrit emails are showing up as being sent late via Yahoo servers
Closed, InvalidPublic

Description

Hi, when i go to my emails to view what i have, i have some gerrit emails. Looking in them shows that i have a code review +2. Clicking link takes me to the patch. But looking at the time. It shows 8am on my email and 12 am on the gerrit code review.

this problem has been happening since gerrit 2.13. Gerrit 2.12 was not a problem. This should be fixed as users who read there emails will get confused on which ones to read as old ones will show as new ones.

Event Timeline

For example this change https://gerrit.wikimedia.org/r/341630 it was merged at 12:05am my time but my email is showing as it being c+2 at 8:31 am my time.

Might want to post the full email headers of the offending email(s)

https://gerrit-review.googlesource.com/ is working in sending the email to my email at the correct times.

gerrit test comment for emails (testing if it affects phabricator too)

Paladox renamed this task from Gerrit emails are showing up as being sent late to Gerrit emails are showing up as being sent late also affecting other service like phabricator.Mar 8 2017, 6:49 PM
Paladox added a project: Phabricator.
Paladox updated the task description. (Show Details)
Paladox added a project: Mail.

I found this in the log on phabricator test instantance in labs /var/log/exim4/mainlog

2017-03-08 06:39:39 1clVG7-0006Co-Td <= root@phabricator.phabricator.eqiad.wmflabs U=root P=local S=1185
2017-03-08 06:39:39 1clVG7-0006Co-Td mx1001.wikimedia.org [2620:0:861:3:208:80:154:76] Network is unreachable
2017-03-08 06:39:39 1clVG7-0006Co-Td ** root@wikimedia.org <root@phabricator.phabricator.eqiad.wmflabs> R=smart_route T=remote_smtp: SMTP error from remote mail server after RCPT TO:<root@wikimedia.org>: host mx1001.wikimedia.org [208.80.154.76]: 550-Verification failed for <root@phabricator.phabricator.eqiad.wmflabs>\n550-Cannot route to remote domain phabricator.phabricator.eqiad.wmflabs\n550 Sender verify failed
2017-03-08 06:39:39 1clVG7-0006Ct-VZ <= <> R=1clVG7-0006Co-Td U=Debian-exim P=local S=2472
2017-03-08 06:39:39 1clVG7-0006Co-Td Completed
2017-03-08 06:39:40 1clVG7-0006Ct-VZ => root@wikimedia.org <root@phabricator.phabricator.eqiad.wmflabs> R=smart_route T=remote_smtp S=2523 H=mx1001.wikimedia.org [208.80.154.76] C="250 OK id=1clVG8-00021n-2G" DT=0s
2017-03-08 06:39:40 1clVG7-0006Ct-VZ Completed

Ottomata subscribed.

Hard to tell if this is a phab/gerrit problem, or just a slow email server problem. Leaving operations tag for now. Maybe @demon can help?

I found this in the log on phabricator test instantance in labs /var/log/exim4/mainlog

2017-03-08 06:39:39 1clVG7-0006Co-Td <= root@phabricator.phabricator.eqiad.wmflabs U=root P=local S=1185
2017-03-08 06:39:39 1clVG7-0006Co-Td mx1001.wikimedia.org [2620:0:861:3:208:80:154:76] Network is unreachable

Well this is unrelated, and expected. Labs should not be able to reach the prod mailserver anyway...

Oh that's an ipv6 address the ipv4 one works.

Either way, we shouldn't be sending labs e-mails via the prod mailserver. I'm not worried about labs at all right now. Trying to see if/what is up with production here...

Also: I'm not seeing any delay on getting Phabricator e-mails. For example: replies to this are in my inbox immediately.

Same with gerrit, getting all my e-mails. I see no production problem on our end

Last point: there's no back up of pending jobs on either gerrit or phabricator. Issue here is either local mail provider or a confusion between labs and production.

Oh, what about merging changes in gerrit?

But how comes email's are showing up as late for me? gerri-review is working perfectly.

I'm pretty sure this is a local issue for you / your ISP, nobody else is seeing this issue.

I have now tested with my outlook account and same problem email taking too long to show up except from the email that was sent asking me to confirm my email for gerrit.

Cannot replicate: I just confirmed a new e-mail within about 5 seconds.

Yes that worked too, but after confirming the email and writing a comment here https://gerrit.wikimedia.org/r/#/c/340900/7 i still have not received the you have a comment email.

Paladox renamed this task from Gerrit emails are showing up as being sent late also affecting other service like phabricator to Gerrit emails are showing up as being sent late.Mar 8 2017, 7:58 PM
Paladox updated the task description. (Show Details)
Paladox removed projects: Phabricator, Upstream.

From P5028 :

Received: from 127.0.0.1  (EHLO mx1001.wikimedia.org) (208.80.154.76)
  by mta1470.mail.gq1.yahoo.com with SMTPS; Wed, 08 Mar 2017 08:31:55 +0000
Received: from cobalt.wikimedia.org ([208.80.154.81]:54510)
	by mx1001.wikimedia.org with esmtp (Exim 4.84_2)
	(envelope-from <gerrit@wikimedia.org>)
	id 1clP6l-0002t4-8w; Wed, 08 Mar 2017 00:05:35 +0000

That sounds to me like Yahoo is throttling emails from mx1001. Given the large number of issues we have with Yahoo's mail servers, I would suggest @Paladox asks them to fix this.

Aklapper renamed this task from Gerrit emails are showing up as being sent late to Gerrit emails are showing up as being sent late via Yahoo servers.Mar 13 2017, 10:33 AM

@valhallasw Hi, yahoo tells me that they doint throttle emails but they do have an internal thing that can block specific domains.

"Yahoo doesn't throttle emails but we do have internal filters in place for security purposes. It does sound like our internal filters may be blocking specific domains. To have this investigated you would need to contact our postmaster team here."

I have filled T160381 as wikimedia will need to fill out a form to get yahoo investigating it further.