Page MenuHomePhabricator

gmail considers all Phabricator email to be spam due to missing SPF record
Closed, ResolvedPublic

Description

I noticed today that gmail has been putting almost all email from phabricator.wikimedia.org into the spam folder. When looking at such mails, I notice the following header:

Received-SPF: fail (google.com: domain of no-reply@phabricator.wikimedia.org does not
        designate 2620:0:861:102:10:64:16:8 as permitted sender)
        client-ip=2620:0:861:102:10:64:16:8;

This would indicate phabricator.wikimedia.org needs a DNS entry specifying an SPF policy that allows mx1001.wikimedia.org to send emails on behalf of it. If I'm understanding correctly, which I might not be.

(Not sure what tags or subsribers are best, so guessing. My apologies for guessing wrong.)

2620:0:861:102:10:64:16:8 has a PTR set to phab1001.eqiad.wmnet.

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 21 2019, 2:07 PM
hashar updated the task description. (Show Details)Feb 21 2019, 2:20 PM
hashar added subscribers: herron, hashar.

I think Keith @herron is the new postmaster!

Some more headers, as requested by Antoine.

Received: from mx1001.wikimedia.org (mx1001.wikimedia.org. [208.80.154.76])
        by mx.google.com with ESMTPS id t5si3777496qta.213.2019.02.21.03.47.27
        for <lwirzenius@wikimedia.org>
        (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256);
        Thu, 21 Feb 2019 03:47:27 -0800 (PST)
Received-SPF: fail (google.com: domain of no-reply@phabricator.wikimedia.org does not
        designate 2620:0:861:102:10:64:16:8 as permitted sender)
        client-ip=2620:0:861:102:10:64:16:8;
Authentication-Results: mx.google.com;
        spf=fail (google.com: domain of no-reply@phabricator.wikimedia.org does
        not designate 2620:0:861:102:10:64:16:8 as permitted sender)
        smtp.mailfrom=no-reply@phabricator.wikimedia.org
Received: from phab1001.eqiad.wmnet ([2620:0:861:102:10:64:16:8]:52030)
        by mx1001.wikimedia.org with esmtp (Exim 4.89)
        (envelope-from <no-reply@phabricator.wikimedia.org>)
        id 1gwmp5-0008Rd-7l
        for lwirzenius@wikimedia.org; Thu, 21 Feb 2019 11:47:27 +0000
Received: from localhost ([::1]:55110 helo=localhost.localdomain)
        by phab1001.eqiad.wmnet with esmtp (Exim 4.84_2)
        (envelope-from <no-reply@phabricator.wikimedia.org>)
        id 1gwmp5-0005e2-2N
        for lwirzenius@wikimedia.org; Thu, 21 Feb 2019 11:47:27 +0000
hashar added a comment.EditedFeb 21 2019, 2:30 PM

From DNS:

$ dig +short TXT phabricator.wikimedia.org
"v=spf1 mx ip4:10.64.32.150 ip6:2620:0:861:103:10:64:32:150 -all"

The IP addresses in the SPF record are for iridium.eqiad.wmnet which has been decommissioned (0c0f94cac1ff428d2e2bb01af0678daa11090fd3 T172487). So I guess it just about updating the entries in DNS. We would need it for both phab1001 and phab2001.

Side question, wouldn't it be sufficient to just whitelist the mx1001 relay instead of each individual servers that might send emails?

Could this be happening to some addresses and not others? I've been checking my most recent Phabricator emails and none of them arrived to spam, and all sfs headers are pass for me. Maybe it is happening only to mails delivered to @wikimedia.org addresses?

ARC-Authentication-Results: i=1; mx.google.com;
       dkim=pass header.i=@wikimedia.org header.s=wikimedia header.b=BZkihoqq;
       spf=pass (google.com: domain of no-reply@phabricator.wikimedia.org designates 208.80.154.76 as permitted sender) smtp.mailfrom=no-reply@phabricator.wikimedia.org;
       dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=wikimedia.org
Return-Path: <no-reply@phabricator.wikimedia.org>
Received: from mx1001.wikimedia.org (mx1001.wikimedia.org. [208.80.154.76])
        by mx.google.com with ESMTPS id e66si950549qka.26.2019.02.21.04.51.05
        for <[redacted]@gmail.com>
        (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256);
        Thu, 21 Feb 2019 04:51:06 -0800 (PST)
Received-SPF: pass (google.com: domain of no-reply@phabricator.wikimedia.org designates 208.80.154.76 as permitted sender) client-ip=208.80.154.76;
Authentication-Results: mx.google.com;
       dkim=pass header.i=@wikimedia.org header.s=wikimedia header.b=BZkihoqq;
       spf=pass (google.com: domain of no-reply@phabricator.wikimedia.org designates 208.80.154.76 as permitted sender) smtp.mailfrom=no-reply@phabricator.wikimedia.org;
       dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=wikimedia.org

Change 491984 had a related patch set uploaded (by Herron; owner: Herron):
[operations/dns@master] phabricator: udpate spf record with phab[12]001 current ipv6 addrs

https://gerrit.wikimedia.org/r/491984

Side question, wouldn't it be sufficient to just whitelist the mx1001 relay instead of each individual servers that might send emails?

Since mx[12]001 are the mxes for phabricator.wikimedia.org they are are included by the 'mx' part of the spf record.

Some more headers, as requested by Antoine.

Received: from mx1001.wikimedia.org (mx1001.wikimedia.org. [208.80.154.76])
        by mx.google.com with ESMTPS id t5si3777496qta.213.2019.02.21.03.47.27
        for <lwirzenius@wikimedia.org>
        (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256);
        Thu, 21 Feb 2019 03:47:27 -0800 (PST)
Received-SPF: fail (google.com: domain of no-reply@phabricator.wikimedia.org does not
        designate 2620:0:861:102:10:64:16:8 as permitted sender)
        client-ip=2620:0:861:102:10:64:16:8;
Authentication-Results: mx.google.com;
        spf=fail (google.com: domain of no-reply@phabricator.wikimedia.org does
        not designate 2620:0:861:102:10:64:16:8 as permitted sender)
        smtp.mailfrom=no-reply@phabricator.wikimedia.org
Received: from phab1001.eqiad.wmnet ([2620:0:861:102:10:64:16:8]:52030)
        by mx1001.wikimedia.org with esmtp (Exim 4.89)
        (envelope-from <no-reply@phabricator.wikimedia.org>)
        id 1gwmp5-0008Rd-7l
        for lwirzenius@wikimedia.org; Thu, 21 Feb 2019 11:47:27 +0000
Received: from localhost ([::1]:55110 helo=localhost.localdomain)
        by phab1001.eqiad.wmnet with esmtp (Exim 4.84_2)
        (envelope-from <no-reply@phabricator.wikimedia.org>)
        id 1gwmp5-0005e2-2N
        for lwirzenius@wikimedia.org; Thu, 21 Feb 2019 11:47:27 +0000

Afaict this is flowing from phab1001 -> mx1001 -> google, yet the spf fail is for the ipv6 address of phab1001. Looks like google is performing spf checks against addresses in the received headers.

So I guess it just about updating the entries in DNS. We would need it for both phab1001 and phab2001.

Yes indeed, adding the current IP addresses of phab[12]001 to the phabricator.wikimedia.org spf record looks to be the right course of action.

Change 491984 merged by Herron:
[operations/dns@master] phabricator: udpate spf record with phab[12]001 current addrs

https://gerrit.wikimedia.org/r/491984

Change 491996 had a related patch set uploaded (by Herron; owner: Herron):
[operations/dns@master] phabricator: udpate spf record with phab[12]001 current addrs

https://gerrit.wikimedia.org/r/491996

Change 491996 merged by Herron:
[operations/dns@master] phabricator: udpate spf record with phab[12]001 current addrs

https://gerrit.wikimedia.org/r/491996

Mentioned in SAL (#wikimedia-operations) [2019-02-21T16:23:53Z] <herron> updated phabricator.wikimedia.org spf record T216714

greg awarded a token.Feb 21 2019, 5:36 PM

@LarsWirzenius maybe this is why you weren't getting phabricator notifications?

@mmodell Yes, that is my conclusion as well. My apologies if I hadn't communicated that.

@LarsWirzenius I'm embarassed now, I didn't notice that it was you who filed the task :-o

herron closed this task as Resolved.Feb 22 2019, 2:52 PM
herron claimed this task.

Looking much better now!

Received-SPF: pass (google.com: domain of no-reply@phabricator.wikimedia.org designates 2620:0:861:102:10:64:16:8 as permitted sender) client-ip=2620:0:861:102:10:64:16:8;
Authentication-Results: mx.google.com;
       spf=pass (google.com: domain of no-reply@phabricator.wikimedia.org designates 2620:0:861:102:10:64:16:8 as permitted sender) smtp.mailfrom=no-reply@phabricator.wikimedia.org
Received: from phab1001.eqiad.wmnet ([2620:0:861:102:10:64:16:8]:57828)
	by mx1001.wikimedia.org with esmtp (Exim 4.89)
	(envelope-from <no-reply@phabricator.wikimedia.org>)
	id 1gxB2W-0001lx-Kb
	for kherron@wikimedia.org; Fri, 22 Feb 2019 13:38:56 +0000
Received: from localhost ([::1]:60836 helo=localhost.localdomain)
	by phab1001.eqiad.wmnet with esmtp (Exim 4.84_2)
	(envelope-from <no-reply@phabricator.wikimedia.org>)
	id 1gxB2W-0000i0-F9; Fri, 22 Feb 2019 13:38:56 +0000

I've not had any Phabricator mail end up in the spam folder since yesterday! So I confirm it seems to work. Thank you!

hashar added a comment.EditedFeb 22 2019, 3:06 PM

antoine-approve

well done!