I've no glue what this means and whether/how this is related to pywikibot or the pywikibot project at pypi

This is just a minor issue that pywikibot-admin account on pypi is currently to no purpose as it can't be activated through e-mail. You use your own account to upload releases, so not a big deal. But for test.pypi.org, this might be a blocker

sudo less /var/log/exim4/mainlog | grep pywikibot:

2020-04-01 12:02:28 1jJc4g-0002aF-7I ** [xqt email] <tools.pywikibot@tools.wmflabs.org> R=dnslookup T=remote_smtp H=[REDACTED] X=TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256 CV=no: SMTP error from remote mail server after pipelined MAIL FROM:<01010171359efc66-6126f5a5-9d4c-435d-b199-28a8c2db379d-000000@ses.pypi.org> SIZE=4599: 550 5.7.1 spf policy (FAILED)
2020-04-01 12:02:29 1jJc4g-0002aF-7I => [multichill email] <tools.pywikibot@tools.wmflabs.org> R=dnslookup T=remote_smtp H=[REDACTED] X=TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256 CV=yes C="250 [REDACTED]"
2020-04-01 12:02:29 1jJc4g-0002aF-7I => [jayvdb email] <tools.pywikibot@tools.wmflabs.org> R=dnslookup T=remote_smtp H=[REDACTED] X=TLS1.2:ECDHE_RSA_CHACHA20_POLY1305:256 CV=yes C="250 [REDACTED]"
2020-04-01 12:02:29 1jJc4g-0002aF-7I -> [legoktm email] <tools.pywikibot@tools.wmflabs.org> R=dnslookup T=remote_smtp H=[REDACTED] X=TLS1.2:ECDHE_RSA_CHACHA20_POLY1305:256 CV=yes C="250 [REDACTED]"
2020-04-01 12:02:29 1jJc4g-0002aF-7I -> [zhuyifei1999 email] <tools.pywikibot@tools.wmflabs.org> R=dnslookup T=remote_smtp H=[REDACTED] X=TLS1.2:ECDHE_RSA_CHACHA20_POLY1305:256 CV=yes C="250 [REDACTED]"
2020-04-01 12:02:29 1jJc4g-0002aF-7I -> [dalba email] <tools.pywikibot@tools.wmflabs.org> R=dnslookup T=remote_smtp H=[REDACTED] X=TLS1.2:ECDHE_RSA_CHACHA20_POLY1305:256 CV=yes C="250 [REDACTED]"
2020-04-01 12:02:29 1jJc4g-0002aF-7I -> [ladsgroup email] <tools.pywikibot@tools.wmflabs.org> R=dnslookup T=remote_smtp H=[REDACTED] X=TLS1.2:ECDHE_RSA_CHACHA20_POLY1305:256 CV=yes C="250 [REDACTED]"
2020-04-01 12:02:29 1jJc4g-0002aF-7I => [valhallasw email] <tools.pywikibot@tools.wmflabs.org> R=dnslookup T=remote_smtp H=[REDACTED] X=TLS1.2:ECDHE_RSA_CHACHA20_POLY1305:256 CV=yes C="250 [REDACTED]"
2020-04-01 12:02:30 1jJc4g-0002aF-7I => [framawiki email] <tools.pywikibot@tools.wmflabs.org> R=dnslookup T=remote_smtp H=[REDACTED] X=TLS1.2:RSA_AES_256_GCM_SHA384:256 CV=yes C="250 [REDACTED]"
2020-04-01 12:02:31 1jJc4g-0002aF-7I ** [dvorapa email] <tools.pywikibot@tools.wmflabs.org> R=dnslookup T=remote_smtp H=[REDACTED] X=TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256 CV=yes: SMTP error from remote mail server after pipelined MAIL FROM:<01010171359efc66-6126f5a5-9d4c-435d-b199-28a8c2db379d-000000@ses.pypi.org> SIZE=4599: 550 5.7.1 Sender Policy Framework of `ses.pypi.org' domain denied your IP address.
2020-04-01 12:02:32 1jJc4g-0002aF-7I => [russblau email] <tools.pywikibot@tools.wmflabs.org> R=dnslookup T=remote_smtp H=[REDACTED] X=TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128 CV=yes C="250 [REDACTED]"

• T120210: tools-mail: check SPF of sender before forwarding email
• T120225: Toolforge: correctly envelope forwarded email
• T104733: Set up A-based SPF for tools.wmflabs.org
• T87338: Mails through deployment-mx SPF & DKIM fails
• T55101: Set up SPF/DKIM for tools.wmflabs.org
• T235401: Emails from discourse-mediawiki.wmflabs.org softfail SPF
• T208281: Set up SPF, DKIM, etc. for new cloud MX servers
• T104871: Mails from tools are being marked as 'spam' by gmail
• T137160: Outgoing mail to wikimedia.org not working on new labs instance
• T53359: tools-mail doesn't deliver mails

@Dvorapa I got the verify mail, I can PM you the link on IRC if you want.

In T249114#6020904, @zhuyifei1999 wrote:

@Dvorapa I got the verify mail, I can PM you the link on IRC if you want.

Not urgent. For pypi.org I don't need it (yet) and for test.pypi.org I can't use it anyway as only @Ladsgroup has access to the repo there.

In T249114#6020953, @Dvorapa wrote:

In T249114#6020904, @zhuyifei1999 wrote:

@Dvorapa I got the verify mail, I can PM you the link on IRC if you want.

Not urgent. For pypi.org I don't need it (yet) and for test.pypi.org I can't use it anyway as only @Ladsgroup has access to the repo there.

Thanks for letting me know. I tried to add pywikibot-admin as the owner but it didn't let me because of this:

ErrorUser 'pywikibot-admin' does not have a verified primary email address and cannot be added as a Owner for project

I think you need to verify it separately. If you want me to add another user, let me know. Thanks.

In T249114#6021297, @Ladsgroup wrote:

ErrorUser 'pywikibot-admin' does not have a verified primary email address and cannot be added as a Owner for project

Yes, that's what this task is about. @zhuyifei1999 If I generate new two e-mails today, would you forward them to my e-mail address (dvorapa~seznam~cz) or activate accounts on both sites using the links in them?

I can forward them when I wake up.

Tommorrow

Forwarded

Thank you! @Ladsgroup could you try add pywikibot-admin to test.pypi.org now?

In T249114#6025057, @Dvorapa wrote:

Thank you! @Ladsgroup could you try add pywikibot-admin to test.pypi.org now?

{{done}}

Thank you!

@zhuyifei1999 Could you once more look for any message(s) from kernel.org? (T245350#6043128)

In T249114#6073828, @Dvorapa wrote:

@zhuyifei1999 Could you once more look for any message(s) from kernel.org? (T245350#6043128)

Don't see it in my inbox. My last email that has anything to do with kernel.org was last December.

That man from btrfs wiki said confirmation e-mail for Pywikibot-test has been sent (up to 14 days ago) :/

@zhuyifei1999 in recent logs I see some emails being forwarded correctly. So indeed the problem is only with the pypi.org domain, right?

root@tools-mail-02:/var/log/exim4# zgrep dvorapa *
mainlog.1:2020-04-27 00:30:23 1jSrf5-0005v1-AT => dvorapa@xxxx.xxx <tools.pywikibot@tools.wmflabs.org> R=dnslookup T=remote_smtp H=mx1.xxxx.xxx [x.x.x.x] X=TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256 CV=yes C="250 2.0.0 Mail 104099816 queued for delivery in session 39f0000001ff."
mainlog.8.gz:2020-04-19 22:24:30 1jQIMG-0006s5-AJ => dvorapa@xxxx.xxx <tools.pywikibot@tools.wmflabs.org> R=dnslookup T=remote_smtp H=mx1.xxxx.xxx [x.x.x.x] X=TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256 CV=yes C="250 2.0.0 Mail 5148014 queued for delivery in session 6d4a00000332."

I think I understand the issue is when this particular thing happens: noreply@pypi.org sends and email to tools.pywikibot@tools.wmflabs.org which fails to reach @Dvorapa personal email server, but reach everyone else.
The error message Sender Policy Framework of 'ses.pypi.org' domain denied your IP address. means to me that we are trying to forward the email from tools-mail-02 but that fails the pypi.org SPF check in the final email server. Makes sense, our toolforge email server is not an allowed sender for the pypi.org domain :-P

After a brief search in google, this seems to be a common problem, which may be solved by using SRS https://en.wikipedia.org/wiki/Sender_Rewriting_Scheme which seems to be an experimental feature in exim: https://github.com/Exim/exim/wiki/SRS

This is to say: I'm not sure how to move forward with this task.

@Dvorapa personal email server

is the no. 1 e-mail provider in Czech & Slovak Republics :D

In T249114#6084919, @Dvorapa wrote:

@Dvorapa personal email server

is the no. 1 e-mail provider in Czech & Slovak Republics :D

Sure, please read my sentence as "server of the provider that @Dvorapa uses for personal email".

Should I contact also them btw?

In T249114#6085686, @Dvorapa wrote:

Should I contact also them btw?

No, at least not about this issue. You ISP is doing the right things to protect you from getting a lot of spam. The issue here is that the Toolforge mail relay is not sending your ISP the right signals to know that instead of being a horrible spammer we are just a relay for messages. @aborrero is starting to investigate how we can send the correct signals so that these kinds of messages work as hoped.

Okay, now I understand, thank you for the explanation.

From what I can tell this is a specific example of the general cross-domain mail forwarding problem that T120225 aims to address

@zhuyifei1999 Sorry to bother you, could you once more look for a btrfs.wiki.kernel.org e-mail? I tried to generate one today. I am unsure if there is a problem with Toolforge or with their wiki (or both)

Same thing as T249114#6073998:

09:09:37 0 ✓ zhuyifei1999@tools-mail-02: ~$ sudo less /var/log/exim4/mainlog | grep pywikibot
2020-05-10 08:22:46 H=ec2-34-211-101-61.us-west-2.compute.amazonaws.com (ip-10-30-118-124.us-west-2.compute.internal) [34.211.101.61] X=TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256 CV=no F=<webapp@ip-10-30-118-124.us-west-2.compute.internal> rejected RCPT <tools.pywikibot@tools.wmflabs.org>: Sender verify failed

I'll also note that btrfs.wiki.kernel.org 's address is also in AWS us-west-2 region.

In T249114#6122268, @zhuyifei1999 wrote:

Same thing as T249114#6073998:

09:09:37 0 ✓ zhuyifei1999@tools-mail-02: ~$ sudo less /var/log/exim4/mainlog | grep pywikibot
2020-05-10 08:22:46 H=ec2-34-211-101-61.us-west-2.compute.amazonaws.com (ip-10-30-118-124.us-west-2.compute.internal) [34.211.101.61] X=TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256 CV=no F=<webapp@ip-10-30-118-124.us-west-2.compute.internal> rejected RCPT <tools.pywikibot@tools.wmflabs.org>: Sender verify failed

I'll also note that btrfs.wiki.kernel.org 's address is also in AWS us-west-2 region.

What does that mean? It seems I'm stuck with a Pywikibot-test account on btrfs wiki, to which I have no access at all. Should I ask btrfs wiki admin to change the e-mail address for the account? (Is that even possible in MediaWiki?)

2020-05-10 08:22:46 H=ec2-34-211-101-61.us-west-2.compute.amazonaws.com (ip-10-30-118-124.us-west-2.compute.internal) [34.211.101.61] Warning: Sender address webapp@ip-10-30-118-124.us-west-2.compute.internal has exceeded rate limit of  messages per 1h
2020-05-10 08:22:46 H=ec2-34-211-101-61.us-west-2.compute.amazonaws.com (ip-10-30-118-124.us-west-2.compute.internal) [34.211.101.61] sender verify fail for <webapp@ip-10-30-118-124.us-west-2.compute.internal>: Unrouteable address
2020-05-10 08:22:46 H=ec2-34-211-101-61.us-west-2.compute.amazonaws.com (ip-10-30-118-124.us-west-2.compute.internal) [34.211.101.61] X=TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256 CV=no F=<webapp@ip-10-30-118-124.us-west-2.compute.internal> rejected RCPT <tools.pywikibot@tools.wmflabs.org>: Sender verify failed

The sender of the email is webapp@ip-10-30-118-124.us-west-2.compute.internal. The hostname of the email address is not valid.

It seems normal for AWS server to have that sort of FQDN internally. This is one of my AWS servers I sometimes test stuffs on:

[ec2-user@ip-172-31-3-[...] ~]$ hostname -f
ip-172-31-3-[...].us-east-2.compute.internal

Whoever maintains kernel wiki installs need to make the server hostname FQDN make sense to the public non-AWS world. We don't relay emails if that email has unknown sender.

I have a conversation with David Sterba from kernel.org wikis. I e-mailed him about the sender hostname, I'll share his reaction when it arrives.

You can refer them to this ticket if needed.

Already did

hey @Dvorapa I suspect this is fixed now..

I just introduced some changes into our mail servers (see T120225: Toolforge: correctly envelope forwarded email) could you please check if you can reproduce now the behavior?