T216714: gmail considers all Phabricator email to be spam due to missing SPF record may be related here

Change 504936 had a related patch set uploaded (by Herron; owner: Herron):
[operations/dns@master] phabricator: remove rfc1918 ip4 addrs from SPF record

https://gerrit.wikimedia.org/r/504936

I am the "User in #2019041710004636" mentioned in the first comment and a very novice subscriber to Phabricator.

Our servers do extensive testing of SPF records, well beyond what's specifically called for by normal SPF processing to detect forgeries, but in the spirit of Getting This Right so that forgeries will as time progresses be detected more reliably.

We reject in many cases where RFC7208 doesn't actually suggest it; when things are broken, this is how we find out.

The SPF record for 'phabricator.wikimedia.org' as it stands at present is

"v=spf1 mx ip4:10.64.16.8 ip6:2620:0:861:102:10:64:16:8 ip4:10.192.32.147 ip6:2620:0:860:103:10:192:32:147 -all"

We reject on e.g. finding RFC1918 addresses in SPF records unless the sender is whitelisted.

AFAICT the IP addresses in the ip4 and ip6 mechanisms are all wrong (anything in the 10.0.0.0/8 CIDR block is certainly wrong), so the IPs should be replaced by those actually used by those servers which send the mail. That seems to be just the mail exchanges, so assuming that their IP addresses are pretty static we can do the lookup from DNS just once, for everybody who will ever want to know, then put the literal addresses in the SPF record :

$ dig +short -t mx phabricator.wikimedia.org
10 mx1001.wikimedia.org.
50 mx2001.wikimedia.org.
$ dig +short -t aaaa mx1001.wikimedia.org
2620:0:861:3:208:80:154:76
$ dig +short -t aaaa mx2001.wikimedia.org
2620:0:860:2:208:80:153:45
$ dig +short -t a mx1001.wikimedia.org
208.80.154.76
$ dig +short -t a mx2001.wikimedia.org
208.80.153.45

The 'mx' mechanism should be removed, it's inefficient and rarely the best option. In any case it should not be placed before the literal IPs.

So the optimal record (zero wasted DNS operations during SPF processing) would be:

"v=spf1 ip6:2620:0:861:3:208:80:154:76 ip6:2620:0:860:2:208:80:153:45 ip4:208.80.154.76 ip4:208.80.153.45 -all"

If the IPv6 addresses currently in the record may legitimtately send mail then they should of course remain in it.

The new record, whatever it is, could be tested at first with '?all' instead of '-all'.

I saw a comment in 504936 about systems checking SPF against 'Received' headers. RFC7208 doesn't permit that - at the time of the SPF checks, no headers have yet been transmitted. I wonder if any sane system does it, but I know there are many Out There which are not sane.

As there are many domains to be maintained within what RFC7208 would probably describe as a single administrative boundary at wikimedia.org, extensive use of the 'redirect' mechanism might be appropriate. This would be a wide-reaching set of changes.

There may be a case for implementing feedback mechanisms as described in RFC6652.

In T221288#5125169, @GedHaywood wrote:

I am the "User in #2019041710004636" mentioned in the first comment and a very novice subscriber to Phabricator.

Thank you for the reports! I'm sorry you couldn't originally be credited by name in the task descriptions as those emails are handled privately.

In T221288#5125169, @GedHaywood wrote:

The 'mx' mechanism should be removed, it's inefficient and rarely the best option. In any case it should not be placed before the literal IPs.

phab[12]001.eqiad.wmnet originate the mail but they relay it all via mx[12]001.wikimedia.org, which also handle incoming mail. The mx item is probably the only one that should really be in our phab SPF record... I'm not sure anyone is too worried about the efficiency of causing receivers to then have to go look up the MX record as well.
For some reason certain providers appear to care about the internal Receive headers too, which is why internal IPs have shown up in the SPF record, but...

In T221288#5125169, @GedHaywood wrote:

I saw a comment in 504936 about systems checking SPF against 'Received' headers. RFC7208 doesn't permit that - at the time of the SPF checks, no headers have yet been transmitted. I wonder if any sane system does it, but I know there are many Out There which are not sane.

Maybe so but if enough providers check that anyway and only one domain (which you've said is set up to perform extra checks also not required by the RFCs) enforces the opposite, I think the larger number of providers, broken though they may be, will have to win until their filtering can be fixed?
Still I'm not convinced by the whole thing. Surely we can't go down the route of adding all mw* servers to the wikimedia.org SPF record for wiki-mail purposes, why would it necessary to add the equivalent for phabricator?
The change above removes the IPv4 internal addresses for those hosts from the SPF record, but the IPv6 addresses remain, even though it sounds like based on the spec alone that shouldn't be necessary?

Change 504936 merged by Herron:
[operations/dns@master] phabricator: remove rfc1918 ip4 addrs from SPF record

https://gerrit.wikimedia.org/r/504936

Is there any further action for this ticket or can we close it?

Do those IPv6 addresses actually send any mail?

If not they can be deleted.

In T221288#5167063, @GedHaywood wrote:

Do those IPv6 addresses actually send any mail?

Yes, these are the IPv6 addresses of the hosts where Phabricator mail originates. Without them listed in the phabricator.wikimedia.org SPF record we observed spf=fail issues, for example https://phabricator.wikimedia.org/T216714#4971926

SPF works on the information in the SMTP envelope.

If mx1001.wikimedia.org has only the IP addresses 208.80.154.76 and 2620:0:861:3:208:80:154:76, then if mx1001.wikimedia.org sends mail to a Google server it will use onf of those to addresses for the connection, and, as per RFC7208, the Google server must only use one of those IPs for any SPF checks which it makes.

Assuming that the headers in the example given are to be believed, in that case the IPv4 address was used for the connection and that is the ONLY address which should have been used for the SPF tests.

Again assuming belief in the headers given, the Google server was wrong to reject mail for an alleged SPF failure based on the presumably internal-only IPv6 address of phab1001.eqiad.wmnet (2620:0:861:102:10:64:16:8), which address appears only in a 'Received' header, and not seen until the SMTP 'DATA' phase - after the envelope has been sent. The header cannot in any case be trusted by the Google server, because it was added by a wmnet server which did not connect to the Google server to send the mail. As far as the Google server's SPF checks are concerned it is part of the random noise in the message.

This _appears_ to be a Google problem. Their server _appears_ to be broken.

I stress appearance and credulity because I find it hard to believe that a Google server could be so broken.

Has Google been approached about this?