SPF for services sending mail from their own domains (gerrit, gitlab, phab) was updated in T366113: Update SPF records as needed

The comment in T355764: Ensure that VRTS (ticket.wikimedia.org) adheres to Google's sender guidelines said:

Since VRTS is sending under <SUPPORT LIST>@wikimedia.org domain, most of what applies to the wikimedia.org domain applies here. Google sees the domain as in compliance with dkim, spf & dmarc.

but the email is actually sent from <SUPPORT LIST>@wikipedia.org

SPF record for wikipedia.org does not include

_cidrs.wikimedia.org

Timing-wise this could also be related to moving outbound email to mx-out{1001,2001}.wikimedia.org (https://gerrit.wikimedia.org/r/1051803)

As a side note, https://wikitech.wikimedia.org/wiki/Sender_Policy_Framework should be updated or removed.

In T369341#9955747, @LSobanski wrote:

The comment in T355764: Ensure that VRTS (ticket.wikimedia.org) adheres to Google's sender guidelines said:

Since VRTS is sending under <SUPPORT LIST>@wikimedia.org domain, most of what applies to the wikimedia.org domain applies here. Google sees the domain as in compliance with dkim, spf & dmarc.

but the email is actually sent from <SUPPORT LIST>@wikipedia.org

SPF record for wikipedia.org does not include

_cidrs.wikimedia.org

This is a bit more complex, queues have return addresses for both <SUPPORT LIST>@wikipedia.org and <SUPPORT LIST>@wikimedia.org, it's not clear to me what defines which one is used. I responded to tickets submitted via mail to both address options and both responses came from the @wikimedia.org address.

The return email is defined in the "System address" field of the queue settings and is set to different values for different queues.

To make things event more interesting I successfully responded to a test ticket from a queue with an @wikipedia.org address.

The list of all used e-mail addresses is defined in VRTS and mirrored to: https://vrt-wiki.wikimedia.org/wiki/List_of_email_addresses

@LSobanski I also just tried to reproduce the error but was unsuccessful. https://ticket.wikimedia.org/otrs/index.pl?Action=AgentTicketZoom;TicketID=13244279;ArticleID=16159676

I sent to a @wikipedia.org address and got a response back from a @wikipedia.org address. 🤔

Since we're unable to reproduce the problem, if anyone is aware of another ticket with a failed delivery it would be helpful to have a link to it here.

I have received the same type of message as listed above for two tickets in checkuser-en-wp queue. (Private information in this queue).

Ticket # 2022030310009117

This is the mail system at host mx-out1001.wikimedia.org.

I'm sorry to have to inform you that your message could not
be delivered to one or more recipients. It's attached below.

For further assistance, please send mail to postmaster.

If you do so, please include this problem report. You can
delete your own text from the attached returned message.

The mail system

<t[redacted]@gmail.com>: host gmail-smtp-in.l.google.com[2607:f8b0:4004:c07::1b]

said: 550-5.7.26 Your email has been blocked because the sender is
unauthenticated. 550-5.7.26 Gmail requires all senders to authenticate with
either SPF or DKIM. 550-5.7.26  550-5.7.26  Authentication results:
550-5.7.26  DKIM = did not pass 550-5.7.26  SPF [wikipedia.org] with ip:
[2620:0:861:1:208:80:154:5] = did not 550-5.7.26 pass 550-5.7.26
550-5.7.26  For instructions on setting up authentication, go to 550 5.7.26
https://support.google.com/mail/answer/81126#authentication
af79cd13be357-79d6933d40csi1349675985a.670 - gsmtp (in reply to end of DATA
command)

(Ticket #2022110210011109)

This is the mail system at host mx-out1001.wikimedia.org.

I'm sorry to have to inform you that your message could not
be delivered to one or more recipients. It's attached below.

For further assistance, please send mail to postmaster.

If you do so, please include this problem report. You can
delete your own text from the attached returned message.

The mail system

<[redacted]@gmail.com>: host

gmail-smtp-in.l.google.com[2607:f8b0:4004:c07::1a] said: 550-5.7.26 Your
email has been blocked because the sender is unauthenticated. 550-5.7.26
Gmail requires all senders to authenticate with either SPF or DKIM.
550-5.7.26  550-5.7.26  Authentication results: 550-5.7.26  DKIM = did not
pass 550-5.7.26  SPF [wikipedia.org] with ip: [2620:0:861:1:208:80:154:5] =
did not 550-5.7.26 pass 550-5.7.26  550-5.7.26  For instructions on setting
up authentication, go to 550 5.7.26
https://support.google.com/mail/answer/81126#authentication
d75a77b69052e-446514c282dsi154267731cf.577 - gsmtp (in reply to end of DATA
command)

Noting that we have identified a further bounced email. (Again, it's from a restricted list with private info attached to the ticket, so many subscribers may not be able to view.) The one thing all of these emails have in common is that they are being sent from the "wikipedia.org" domain. Ensuring that domain has the proper SPF/DKIM authentications seems to be the straightforward solution.

See below - ticket 2024070610005364

This is the mail system at host mx-out1001.wikimedia.org.

I'm sorry to have to inform you that your message could not
be delivered to one or more recipients. It's attached below.

For further assistance, please send mail to postmaster.

If you do so, please include this problem report. You can
delete your own text from the attached returned message.

The mail system

<[redacted]@gmail.com>: host gmail-smtp-in.l.google.com[172.253.62.26] said:

550-5.7.26 Your email has been blocked because the sender is
unauthenticated. 550-5.7.26 Gmail requires all senders to authenticate with
either SPF or DKIM. 550-5.7.26  550-5.7.26  Authentication results:
550-5.7.26  DKIM = did not pass 550-5.7.26  SPF [wikipedia.org] with ip:
[208.80.154.5] = did not pass 550-5.7.26  550-5.7.26  For instructions on
setting up authentication, go to 550 5.7.26
https://support.google.com/mail/answer/81126#authentication
6a1803df08f44-6b603d4d529si15768836d6.337 - gsmtp (in reply to end of DATA
command)

https://ticket.wikimedia.org/otrs/index.pl?Action=AgentTicketZoom;TicketID=13245515#16161333

another bounceback

This is the mail system at host mx-out1001.wikimedia.org.

I'm sorry to have to inform you that your message could not
be delivered to one or more recipients. It's attached below.

For further assistance, please send mail to postmaster.

If you do so, please include this problem report. You can
delete your own text from the attached returned message.

The mail system

<redacted@gmail.com>: host gmail-smtp-in.l.google.com[172.253.62.26] said:

550-5.7.26 Your email has been blocked because the sender is
unauthenticated. 550-5.7.26 Gmail requires all senders to authenticate with
either SPF or DKIM. 550-5.7.26  550-5.7.26  Authentication results:
550-5.7.26  DKIM = did not pass 550-5.7.26  SPF [wikipedia.org] with ip:
[208.80.154.5] = did not pass 550-5.7.26  550-5.7.26  For instructions on
setting up authentication, go to 550 5.7.26
https://support.google.com/mail/answer/81126#authentication
6a1803df08f44-6b603d4d529si15768836d6.337 - gsmtp (in reply to end of DATA
command)

https://ticket.wikimedia.org/otrs/index.pl?Action=AgentTicketZoom;TicketID=13245660

another bounceback

This is the mail system at host mx-out1001.wikimedia.org.

I'm sorry to have to inform you that your message could not
be delivered to one or more recipients. It's attached below.

For further assistance, please send mail to postmaster.

If you do so, please include this problem report. You can
delete your own text from the attached returned message.

The mail system

<[redacted]@gmail.com>: host gmail-smtp-in.l.google.com[172.253.62.27] said:

550-5.7.26 The MAIL FROM domain [wikipedia.org] has an SPF record with a
hard 550-5.7.26 fail policy (-all) but it fails to pass SPF checks with the
ip: 550-5.7.26 [208.80.154.5]. To best protect our users from spam and
phishing, 550-5.7.26 the message has been blocked. For instructions on
setting up 550-5.7.26 authentication, go to 550 5.7.26
https://support.google.com/mail/answer/81126#authentication
d75a77b69052e-447db46a8f8si34261251cf.594 - gsmtp (in reply to end of DATA
command)

This message may not always be caused by what it says, since Gmail often also takes some other factors into account.

However, in this case this does fail SPF. (Return-Path: <info-en(aŧ)wikipedia.org>), and the SPF record is that no IP sends email with this domain:

wikipedia.org.		600	IN	TXT	"v=spf1 -all"

Moreover, there is no DKIM signature.

Note that for wikimedia.org, they do pass SPF and use DKIM

If this was working before, that was probably just because the old mx{1001,2001}.wikimedia.org had an high-enough reputation (and maybe they DKIM-signed the wikipedia.org emails?), and then broke when changing outbound servers on T365395

@LSobanski: see the test ticket https://ticket.wikimedia.org/otrs/index.pl?Action=AgentTicketZoom;TicketID=13245774

The sending address depends on the Queue. If you send from info-en, that goes out from info-en at wikiMedia, but if it was on info-en::vandalism, that goes out from info-en at wikiPedia. Some other queues and subqueues are probably using a wikipedia.org domain as well, this is just the first case I found.

Adding SPF and DKIM for wikipedia.org should fix the issue.

I'll prep a patch to fix the SPF record, I was not aware mail was being *sent* from the wikipedia.org domain. I'll also take a look and see if I can query for all the sender addresses from the database.

Change #1052768 had a related patch set uploaded (by JHathaway; author: JHathaway):

[operations/dns@master] wikipedia.org spf: indicate mail is sent from this domain.

https://gerrit.wikimedia.org/r/1052768

Change #1052768 merged by JHathaway:

[operations/dns@master] wikipedia.org spf: indicate mail is sent from this domain.

https://gerrit.wikimedia.org/r/1052768

The spf record has been updated:

wikipedia.org.		577	IN	TXT	"v=spf1 include:_cidrs.wikimedia.org ~all"

please let me know if the errors persist

Change #1052792 had a related patch set uploaded (by JHathaway; author: JHathaway):

[operations/dns@master] wikipedia.org spf: add a comment

https://gerrit.wikimedia.org/r/1052792

Change #1052792 merged by JHathaway:

[operations/dns@master] wikipedia.org spf: add a comment

https://gerrit.wikimedia.org/r/1052792