Page MenuHomePhabricator

E-mails from noreply@pypi.org to tools.pywikibot@tools.wmflabs.org are not forwarded to certain recipients due to SPF
Closed, ResolvedPublic

Description

Pywikibot has issues with e-mails using tool account: https://github.com/pypa/pypi-support/issues/297

Those e-mails (e-mail address verification) using tools.pywikibot@tools.wmflabs.org were not delivered to tool maintainers.

Even today (13 PM UTC+1) I got no e-mail I requested by clicking the button in pypi.org account.

More info in upstream issue (e-mail seems delivered from pypi.org side)

Event Timeline

Dvorapa created this task.Apr 1 2020, 2:24 PM
Restricted Application edited projects, added cloud-services-team (Kanban); removed cloud-services-team. · View Herald TranscriptApr 1 2020, 2:24 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Dvorapa updated the task description. (Show Details)Apr 1 2020, 2:25 PM
Dvorapa edited projects, added Cloud-Services; removed Toolforge.
Dvorapa edited projects, added Toolforge; removed Cloud-Services.
Restricted Application added a subscriber: pywikibot-bugs-list. · View Herald TranscriptApr 1 2020, 2:30 PM
bd808 renamed this task from E-mails from noreply@pypi.org are not delivered for more than a year to E-mails from noreply@pypi.org are not delivered to tools.pywikibot@tools.wmflabs.org for more than a year.Apr 1 2020, 2:30 PM
Xqt added a subscriber: Xqt.Apr 1 2020, 3:32 PM

I've no glue what this means and whether/how this is related to pywikibot or the pywikibot project at pypi

Dvorapa added a comment.EditedApr 1 2020, 3:37 PM

This is just a minor issue that pywikibot-admin account on pypi is currently to no purpose as it can't be activated through e-mail. You use your own account to upload releases, so not a big deal. But for test.pypi.org, this might be a blocker

sudo less /var/log/exim4/mainlog | grep pywikibot:

2020-04-01 12:02:28 1jJc4g-0002aF-7I ** [xqt email] <tools.pywikibot@tools.wmflabs.org> R=dnslookup T=remote_smtp H=[REDACTED] X=TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256 CV=no: SMTP error from remote mail server after pipelined MAIL FROM:<01010171359efc66-6126f5a5-9d4c-435d-b199-28a8c2db379d-000000@ses.pypi.org> SIZE=4599: 550 5.7.1 spf policy (FAILED)
2020-04-01 12:02:29 1jJc4g-0002aF-7I => [multichill email] <tools.pywikibot@tools.wmflabs.org> R=dnslookup T=remote_smtp H=[REDACTED] X=TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256 CV=yes C="250 [REDACTED]"
2020-04-01 12:02:29 1jJc4g-0002aF-7I => [jayvdb email] <tools.pywikibot@tools.wmflabs.org> R=dnslookup T=remote_smtp H=[REDACTED] X=TLS1.2:ECDHE_RSA_CHACHA20_POLY1305:256 CV=yes C="250 [REDACTED]"
2020-04-01 12:02:29 1jJc4g-0002aF-7I -> [legoktm email] <tools.pywikibot@tools.wmflabs.org> R=dnslookup T=remote_smtp H=[REDACTED] X=TLS1.2:ECDHE_RSA_CHACHA20_POLY1305:256 CV=yes C="250 [REDACTED]"
2020-04-01 12:02:29 1jJc4g-0002aF-7I -> [zhuyifei1999 email] <tools.pywikibot@tools.wmflabs.org> R=dnslookup T=remote_smtp H=[REDACTED] X=TLS1.2:ECDHE_RSA_CHACHA20_POLY1305:256 CV=yes C="250 [REDACTED]"
2020-04-01 12:02:29 1jJc4g-0002aF-7I -> [dalba email] <tools.pywikibot@tools.wmflabs.org> R=dnslookup T=remote_smtp H=[REDACTED] X=TLS1.2:ECDHE_RSA_CHACHA20_POLY1305:256 CV=yes C="250 [REDACTED]"
2020-04-01 12:02:29 1jJc4g-0002aF-7I -> [ladsgroup email] <tools.pywikibot@tools.wmflabs.org> R=dnslookup T=remote_smtp H=[REDACTED] X=TLS1.2:ECDHE_RSA_CHACHA20_POLY1305:256 CV=yes C="250 [REDACTED]"
2020-04-01 12:02:29 1jJc4g-0002aF-7I => [valhallasw email] <tools.pywikibot@tools.wmflabs.org> R=dnslookup T=remote_smtp H=[REDACTED] X=TLS1.2:ECDHE_RSA_CHACHA20_POLY1305:256 CV=yes C="250 [REDACTED]"
2020-04-01 12:02:30 1jJc4g-0002aF-7I => [framawiki email] <tools.pywikibot@tools.wmflabs.org> R=dnslookup T=remote_smtp H=[REDACTED] X=TLS1.2:RSA_AES_256_GCM_SHA384:256 CV=yes C="250 [REDACTED]"
2020-04-01 12:02:31 1jJc4g-0002aF-7I ** [dvorapa email] <tools.pywikibot@tools.wmflabs.org> R=dnslookup T=remote_smtp H=[REDACTED] X=TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256 CV=yes: SMTP error from remote mail server after pipelined MAIL FROM:<01010171359efc66-6126f5a5-9d4c-435d-b199-28a8c2db379d-000000@ses.pypi.org> SIZE=4599: 550 5.7.1 Sender Policy Framework of `ses.pypi.org' domain denied your IP address.
2020-04-01 12:02:32 1jJc4g-0002aF-7I => [russblau email] <tools.pywikibot@tools.wmflabs.org> R=dnslookup T=remote_smtp H=[REDACTED] X=TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128 CV=yes C="250 [REDACTED]"
zhuyifei1999 renamed this task from E-mails from noreply@pypi.org are not delivered to tools.pywikibot@tools.wmflabs.org for more than a year to E-mails from noreply@pypi.org to tools.pywikibot@tools.wmflabs.org are not forwarded to certain recipients due to SPF.Apr 1 2020, 9:08 PM

@Dvorapa I got the verify mail, I can PM you the link on IRC if you want.

Dvorapa updated the task description. (Show Details)Apr 1 2020, 9:50 PM

@Dvorapa I got the verify mail, I can PM you the link on IRC if you want.

Not urgent. For pypi.org I don't need it (yet) and for test.pypi.org I can't use it anyway as only @Ladsgroup has access to the repo there.

@Dvorapa I got the verify mail, I can PM you the link on IRC if you want.

Not urgent. For pypi.org I don't need it (yet) and for test.pypi.org I can't use it anyway as only @Ladsgroup has access to the repo there.

Thanks for letting me know. I tried to add pywikibot-admin as the owner but it didn't let me because of this:

ErrorUser 'pywikibot-admin' does not have a verified primary email address and cannot be added as a Owner for project

I think you need to verify it separately. If you want me to add another user, let me know. Thanks.

ErrorUser 'pywikibot-admin' does not have a verified primary email address and cannot be added as a Owner for project

Yes, that's what this task is about. @zhuyifei1999 If I generate new two e-mails today, would you forward them to my e-mail address (dvorapa~seznam~cz) or activate accounts on both sites using the links in them?

I can forward them when I wake up.

Thank you! @Ladsgroup could you try add pywikibot-admin to test.pypi.org now?

Thank you! @Ladsgroup could you try add pywikibot-admin to test.pypi.org now?

{{done}}

@zhuyifei1999 Could you once more look for any message(s) from kernel.org? (T245350#6043128)

@zhuyifei1999 Could you once more look for any message(s) from kernel.org? (T245350#6043128)

Don't see it in my inbox. My last email that has anything to do with kernel.org was last December.

01:37:33 0 ✓ zhuyifei1999@tools-mail-02: ~$ sudo less /var/log/exim4/mainlog | grep pywikibot
2020-04-20 15:24:48 H=ec2-34-211-101-61.us-west-2.compute.amazonaws.com (ip-10-30-118-118.us-west-2.compute.internal) [34.211.101.61] X=TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256 CV=no F=<webapp@ip-10-30-118-118.us-west-2.compute.internal> rejected RCPT <tools.pywikibot@tools.wmflabs.org>: Sender verify failed

That man from btrfs wiki said confirmation e-mail for Pywikibot-test has been sent (up to 14 days ago) :/

JHedden triaged this task as Medium priority.Apr 21 2020, 4:19 PM
JHedden raised the priority of this task from Medium to Needs Triage.
JHedden moved this task from Inbox to Soon! on the cloud-services-team (Kanban) board.
JHedden triaged this task as Medium priority.Apr 21 2020, 4:27 PM
aborrero added a subscriber: aborrero.EditedApr 27 2020, 1:07 PM

@zhuyifei1999 in recent logs I see some emails being forwarded correctly. So indeed the problem is only with the pypi.org domain, right?

root@tools-mail-02:/var/log/exim4# zgrep dvorapa *
mainlog.1:2020-04-27 00:30:23 1jSrf5-0005v1-AT => dvorapa@xxxx.xxx <tools.pywikibot@tools.wmflabs.org> R=dnslookup T=remote_smtp H=mx1.xxxx.xxx [x.x.x.x] X=TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256 CV=yes C="250 2.0.0 Mail 104099816 queued for delivery in session 39f0000001ff."
mainlog.8.gz:2020-04-19 22:24:30 1jQIMG-0006s5-AJ => dvorapa@xxxx.xxx <tools.pywikibot@tools.wmflabs.org> R=dnslookup T=remote_smtp H=mx1.xxxx.xxx [x.x.x.x] X=TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256 CV=yes C="250 2.0.0 Mail 5148014 queued for delivery in session 6d4a00000332."

I think I understand the issue is when this particular thing happens: noreply@pypi.org sends and email to tools.pywikibot@tools.wmflabs.org which fails to reach @Dvorapa personal email server, but reach everyone else.
The error message Sender Policy Framework of 'ses.pypi.org' domain denied your IP address. means to me that we are trying to forward the email from tools-mail-02 but that fails the pypi.org SPF check in the final email server. Makes sense, our toolforge email server is not an allowed sender for the pypi.org domain :-P

After a brief search in google, this seems to be a common problem, which may be solved by using SRS https://en.wikipedia.org/wiki/Sender_Rewriting_Scheme which seems to be an experimental feature in exim: https://github.com/Exim/exim/wiki/SRS

This is to say: I'm not sure how to move forward with this task.

@Dvorapa personal email server

is the no. 1 e-mail provider in Czech & Slovak Republics :D

@Dvorapa personal email server

is the no. 1 e-mail provider in Czech & Slovak Republics :D

Sure, please read my sentence as "server of the provider that @Dvorapa uses for personal email".

Should I contact also them btw?

Should I contact also them btw?

No, at least not about this issue. You ISP is doing the right things to protect you from getting a lot of spam. The issue here is that the Toolforge mail relay is not sending your ISP the right signals to know that instead of being a horrible spammer we are just a relay for messages. @aborrero is starting to investigate how we can send the correct signals so that these kinds of messages work as hoped.

Okay, now I understand, thank you for the explanation.

herron added a subscriber: herron.May 4 2020, 10:44 PM

From what I can tell this is a specific example of the general cross-domain mail forwarding problem that T120225 aims to address

@zhuyifei1999 Sorry to bother you, could you once more look for a btrfs.wiki.kernel.org e-mail? I tried to generate one today. I am unsure if there is a problem with Toolforge or with their wiki (or both)

Same thing as T249114#6073998:

09:09:37 0 ✓ zhuyifei1999@tools-mail-02: ~$ sudo less /var/log/exim4/mainlog | grep pywikibot
2020-05-10 08:22:46 H=ec2-34-211-101-61.us-west-2.compute.amazonaws.com (ip-10-30-118-124.us-west-2.compute.internal) [34.211.101.61] X=TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256 CV=no F=<webapp@ip-10-30-118-124.us-west-2.compute.internal> rejected RCPT <tools.pywikibot@tools.wmflabs.org>: Sender verify failed

I'll also note that btrfs.wiki.kernel.org 's address is also in AWS us-west-2 region.

Same thing as T249114#6073998:

09:09:37 0 ✓ zhuyifei1999@tools-mail-02: ~$ sudo less /var/log/exim4/mainlog | grep pywikibot
2020-05-10 08:22:46 H=ec2-34-211-101-61.us-west-2.compute.amazonaws.com (ip-10-30-118-124.us-west-2.compute.internal) [34.211.101.61] X=TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256 CV=no F=<webapp@ip-10-30-118-124.us-west-2.compute.internal> rejected RCPT <tools.pywikibot@tools.wmflabs.org>: Sender verify failed

I'll also note that btrfs.wiki.kernel.org 's address is also in AWS us-west-2 region.

What does that mean? It seems I'm stuck with a Pywikibot-test account on btrfs wiki, to which I have no access at all. Should I ask btrfs wiki admin to change the e-mail address for the account? (Is that even possible in MediaWiki?)

2020-05-10 08:22:46 H=ec2-34-211-101-61.us-west-2.compute.amazonaws.com (ip-10-30-118-124.us-west-2.compute.internal) [34.211.101.61] Warning: Sender address webapp@ip-10-30-118-124.us-west-2.compute.internal has exceeded rate limit of  messages per 1h
2020-05-10 08:22:46 H=ec2-34-211-101-61.us-west-2.compute.amazonaws.com (ip-10-30-118-124.us-west-2.compute.internal) [34.211.101.61] sender verify fail for <webapp@ip-10-30-118-124.us-west-2.compute.internal>: Unrouteable address
2020-05-10 08:22:46 H=ec2-34-211-101-61.us-west-2.compute.amazonaws.com (ip-10-30-118-124.us-west-2.compute.internal) [34.211.101.61] X=TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256 CV=no F=<webapp@ip-10-30-118-124.us-west-2.compute.internal> rejected RCPT <tools.pywikibot@tools.wmflabs.org>: Sender verify failed

The sender of the email is webapp@ip-10-30-118-124.us-west-2.compute.internal. The hostname of the email address is not valid.

It seems normal for AWS server to have that sort of FQDN internally. This is one of my AWS servers I sometimes test stuffs on:

[ec2-user@ip-172-31-3-[...] ~]$ hostname -f
ip-172-31-3-[...].us-east-2.compute.internal

Whoever maintains kernel wiki installs need to make the server hostname FQDN make sense to the public non-AWS world. We don't relay emails if that email has unknown sender.

Dvorapa added a comment.EditedMay 10 2020, 12:22 PM

I have a conversation with David Sterba from kernel.org wikis. I e-mailed him about the sender hostname, I'll share his reaction when it arrives.

You can refer them to this ticket if needed.

hey @Dvorapa I suspect this is fixed now..

I just introduced some changes into our mail servers (see T120225: Toolforge: correctly envelope forwarded email) could you please check if you can reproduce now the behavior?

aborrero closed this task as Resolved.Thu, Jul 2, 11:51 AM
aborrero claimed this task.

Closing task now. Please feel free to reopen if required.