gmail considers all Phabricator email to be spam due to missing SPF record
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	• LarsWirzenius
	Feb 21 2019, 2:07 PM

Description

I noticed today that gmail has been putting almost all email from phabricator.wikimedia.org into the spam folder. When looking at such mails, I notice the following header:

Received-SPF: fail (google.com: domain of no-reply@phabricator.wikimedia.org does not
        designate 2620:0:861:102:10:64:16:8 as permitted sender)
        client-ip=2620:0:861:102:10:64:16:8;

This would indicate phabricator.wikimedia.org needs a DNS entry specifying an SPF policy that allows mx1001.wikimedia.org to send emails on behalf of it. If I'm understanding correctly, which I might not be.

(Not sure what tags or subsribers are best, so guessing. My apologies for guessing wrong.)

2620:0:861:102:10:64:16:8 has a PTR set to phab1001.eqiad.wmnet.

Details

	Subject	Repo	Branch	Lines +/-
	phabricator: udpate spf record with phab[12]001 current addrs	operations/dns	master	+1 -1
	phabricator: udpate spf record with phab[12]001 current addrs	operations/dns	master	+1 -1

Customize query in gerrit

Related Objects

Mentioned In: T221288: Phabricator SPF record contains internal addressing for phab[12]001
T144381: "Wikimedia Foundation mail couldn't verify phabricator.wikimedia.org actually sent this message (and not a spammer)" in GMail
T116805: DomainKeys Identified Mail (DKIM) for phabricator.wikimedia.org
Mentioned Here: rODNS0c0f94cac1ff: decom iridium
T172487: decom iridium

Event Timeline

• LarsWirzenius created this task.Feb 21 2019, 2:07 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 21 2019, 2:07 PM

hashar updated the task description. (Show Details)Feb 21 2019, 2:20 PM

I think Keith @herron is the new postmaster!

Some more headers, as requested by Antoine.

Received: from mx1001.wikimedia.org (mx1001.wikimedia.org. [208.80.154.76])
        by mx.google.com with ESMTPS id t5si3777496qta.213.2019.02.21.03.47.27
        for <lwirzenius@wikimedia.org>
        (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256);
        Thu, 21 Feb 2019 03:47:27 -0800 (PST)
Received-SPF: fail (google.com: domain of no-reply@phabricator.wikimedia.org does not
        designate 2620:0:861:102:10:64:16:8 as permitted sender)
        client-ip=2620:0:861:102:10:64:16:8;
Authentication-Results: mx.google.com;
        spf=fail (google.com: domain of no-reply@phabricator.wikimedia.org does
        not designate 2620:0:861:102:10:64:16:8 as permitted sender)
        smtp.mailfrom=no-reply@phabricator.wikimedia.org
Received: from phab1001.eqiad.wmnet ([2620:0:861:102:10:64:16:8]:52030)
        by mx1001.wikimedia.org with esmtp (Exim 4.89)
        (envelope-from <no-reply@phabricator.wikimedia.org>)
        id 1gwmp5-0008Rd-7l
        for lwirzenius@wikimedia.org; Thu, 21 Feb 2019 11:47:27 +0000
Received: from localhost ([::1]:55110 helo=localhost.localdomain)
        by phab1001.eqiad.wmnet with esmtp (Exim 4.84_2)
        (envelope-from <no-reply@phabricator.wikimedia.org>)
        id 1gwmp5-0005e2-2N
        for lwirzenius@wikimedia.org; Thu, 21 Feb 2019 11:47:27 +0000

From DNS:

$ dig +short TXT phabricator.wikimedia.org
"v=spf1 mx ip4:10.64.32.150 ip6:2620:0:861:103:10:64:32:150 -all"

The IP addresses in the SPF record are for iridium.eqiad.wmnet which has been decommissioned (0c0f94cac1ff428d2e2bb01af0678daa11090fd3 T172487). So I guess it just about updating the entries in DNS. We would need it for both phab1001 and phab2001.

Side question, wouldn't it be sufficient to just whitelist the mx1001 relay instead of each individual servers that might send emails?

Could this be happening to some addresses and not others? I've been checking my most recent Phabricator emails and none of them arrived to spam, and all sfs headers are pass for me. Maybe it is happening only to mails delivered to @wikimedia.org addresses?

ARC-Authentication-Results: i=1; mx.google.com;
       dkim=pass header.i=@wikimedia.org header.s=wikimedia header.b=BZkihoqq;
       spf=pass (google.com: domain of no-reply@phabricator.wikimedia.org designates 208.80.154.76 as permitted sender) smtp.mailfrom=no-reply@phabricator.wikimedia.org;
       dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=wikimedia.org
Return-Path: <no-reply@phabricator.wikimedia.org>
Received: from mx1001.wikimedia.org (mx1001.wikimedia.org. [208.80.154.76])
        by mx.google.com with ESMTPS id e66si950549qka.26.2019.02.21.04.51.05
        for <[redacted]@gmail.com>
        (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256);
        Thu, 21 Feb 2019 04:51:06 -0800 (PST)
Received-SPF: pass (google.com: domain of no-reply@phabricator.wikimedia.org designates 208.80.154.76 as permitted sender) client-ip=208.80.154.76;
Authentication-Results: mx.google.com;
       dkim=pass header.i=@wikimedia.org header.s=wikimedia header.b=BZkihoqq;
       spf=pass (google.com: domain of no-reply@phabricator.wikimedia.org designates 208.80.154.76 as permitted sender) smtp.mailfrom=no-reply@phabricator.wikimedia.org;
       dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=wikimedia.org

Change 491984 had a related patch set uploaded (by Herron; owner: Herron):
[operations/dns@master] phabricator: udpate spf record with phab[12]001 current ipv6 addrs

https://gerrit.wikimedia.org/r/491984

gerritbot added a project: Patch-For-Review.Feb 21 2019, 3:18 PM

In T216714#4971931, @hashar wrote:

Side question, wouldn't it be sufficient to just whitelist the mx1001 relay instead of each individual servers that might send emails?

Since mx[12]001 are the mxes for phabricator.wikimedia.org they are are included by the 'mx' part of the spf record.

In T216714#4971926, @LarsWirzenius wrote:

Some more headers, as requested by Antoine.

Received: from mx1001.wikimedia.org (mx1001.wikimedia.org. [208.80.154.76])
        by mx.google.com with ESMTPS id t5si3777496qta.213.2019.02.21.03.47.27
        for <lwirzenius@wikimedia.org>
        (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256);
        Thu, 21 Feb 2019 03:47:27 -0800 (PST)
Received-SPF: fail (google.com: domain of no-reply@phabricator.wikimedia.org does not
        designate 2620:0:861:102:10:64:16:8 as permitted sender)
        client-ip=2620:0:861:102:10:64:16:8;
Authentication-Results: mx.google.com;
        spf=fail (google.com: domain of no-reply@phabricator.wikimedia.org does
        not designate 2620:0:861:102:10:64:16:8 as permitted sender)
        smtp.mailfrom=no-reply@phabricator.wikimedia.org
Received: from phab1001.eqiad.wmnet ([2620:0:861:102:10:64:16:8]:52030)
        by mx1001.wikimedia.org with esmtp (Exim 4.89)
        (envelope-from <no-reply@phabricator.wikimedia.org>)
        id 1gwmp5-0008Rd-7l
        for lwirzenius@wikimedia.org; Thu, 21 Feb 2019 11:47:27 +0000
Received: from localhost ([::1]:55110 helo=localhost.localdomain)
        by phab1001.eqiad.wmnet with esmtp (Exim 4.84_2)
        (envelope-from <no-reply@phabricator.wikimedia.org>)
        id 1gwmp5-0005e2-2N
        for lwirzenius@wikimedia.org; Thu, 21 Feb 2019 11:47:27 +0000

Afaict this is flowing from phab1001 -> mx1001 -> google, yet the spf fail is for the ipv6 address of phab1001. Looks like google is performing spf checks against addresses in the received headers.

In T216714#4971931, @hashar wrote:

So I guess it just about updating the entries in DNS. We would need it for both phab1001 and phab2001.

Yes indeed, adding the current IP addresses of phab[12]001 to the phabricator.wikimedia.org spf record looks to be the right course of action.

Change 491984 merged by Herron:
[operations/dns@master] phabricator: udpate spf record with phab[12]001 current addrs

https://gerrit.wikimedia.org/r/491984

Change 491996 had a related patch set uploaded (by Herron; owner: Herron):
[operations/dns@master] phabricator: udpate spf record with phab[12]001 current addrs

https://gerrit.wikimedia.org/r/491996

Change 491996 merged by Herron:
[operations/dns@master] phabricator: udpate spf record with phab[12]001 current addrs

https://gerrit.wikimedia.org/r/491996

Mentioned in SAL (#wikimedia-operations) [2019-02-21T16:23:53Z] <herron> updated phabricator.wikimedia.org spf record T216714

greg awarded a token.Feb 21 2019, 5:36 PM

@LarsWirzenius maybe this is why you weren't getting phabricator notifications?

@mmodell Yes, that is my conclusion as well. My apologies if I hadn't communicated that.

@LarsWirzenius I'm embarassed now, I didn't notice that it was you who filed the task :-o

Looking much better now!

Received-SPF: pass (google.com: domain of no-reply@phabricator.wikimedia.org designates 2620:0:861:102:10:64:16:8 as permitted sender) client-ip=2620:0:861:102:10:64:16:8;
Authentication-Results: mx.google.com;
       spf=pass (google.com: domain of no-reply@phabricator.wikimedia.org designates 2620:0:861:102:10:64:16:8 as permitted sender) smtp.mailfrom=no-reply@phabricator.wikimedia.org
Received: from phab1001.eqiad.wmnet ([2620:0:861:102:10:64:16:8]:57828)
	by mx1001.wikimedia.org with esmtp (Exim 4.89)
	(envelope-from <no-reply@phabricator.wikimedia.org>)
	id 1gxB2W-0001lx-Kb
	for kherron@wikimedia.org; Fri, 22 Feb 2019 13:38:56 +0000
Received: from localhost ([::1]:60836 helo=localhost.localdomain)
	by phab1001.eqiad.wmnet with esmtp (Exim 4.84_2)
	(envelope-from <no-reply@phabricator.wikimedia.org>)
	id 1gxB2W-0000i0-F9; Fri, 22 Feb 2019 13:38:56 +0000

I've not had any Phabricator mail end up in the spam folder since yesterday! So I confirm it seems to work. Thank you!

• LarsWirzenius awarded a token.Feb 22 2019, 2:59 PM