Page MenuHomePhabricator
Create Task
Maniphest T120225

Toolforge: correctly envelope forwarded email
Closed, ResolvedPublic
Actions

Description

The current settings cause the SPF record of the original from: host to be checked, rather than those of tools.wmflabs.org:

valhallasw@gmail.com -> valhallasw@tools.wmflabs.org -> valhallasw@arctus.nl

Delivered-To: valhallasw@arctus.nl
Received: by 10.107.55.7 with SMTP id e7csp52297ioa;
        Thu, 3 Dec 2015 07:29:28 -0800 (PST)
X-Received: by 10.140.232.65 with SMTP id d62mr12365942qhc.27.1449156568542;
        Thu, 03 Dec 2015 07:29:28 -0800 (PST)
Return-Path: <valhallasw@gmail.com>
Received: from mail.tools.wmflabs.org (mail.tools.wmflabs.org. [208.80.155.162])
        by mx.google.com with ESMTPS id w104si1741458qge.65.2015.12.03.07.29.28
        for <valhallasw@arctus.nl>
        (version=TLS1 cipher=AES128-SHA bits=128/128);
        Thu, 03 Dec 2015 07:29:28 -0800 (PST)
Received-SPF: softfail (google.com: domain of transitioning valhallasw@gmail.com does not designate 208.80.155.162 as permitted sender) client-ip=208.80.155.162;
Authentication-Results: mx.google.com;
       spf=softfail (google.com: domain of transitioning valhallasw@gmail.com does not designate 208.80.155.162 as permitted sender) smtp.mailfrom=valhallasw@gmail.com;
       dkim=pass header.i=@gmail.com;
       dmarc=pass (p=NONE dis=NONE) header.from=gmail.com
Received: from mail-ig0-f174.google.com ([209.85.213.174])
	by mail.tools.wmflabs.org with esmtp (Exim 4.76)
	(envelope-from <valhallasw@gmail.com>)
	id 1a4Vp2-0001sC-6p
	for valhallasw@tools.wmflabs.org; Thu, 03 Dec 2015 15:29:28 +0000
Received: by igbxm8 with SMTP id xm8so14673378igb.1
        for <valhallasw@tools.wmflabs.org>; Thu, 03 Dec 2015 07:29:20 -0800 (PST)

Details

SubjectRepoBranchLines +/-
toolforge: mailrelay: introduce SRS to correctly envelope forwarded emailsoperations/puppetproduction+51 -2
toolforge: mailrelay: migrate TLS cert to acme-chiefoperations/puppetproduction+7 -27
toolforge: mailrelay: introduce support for disabling TLSoperations/puppetproduction+24 -19
Customize query in gerrit

Related Objects
Search...

Event Timeline

valhallasw created this task.Dec 3 2015, 3:34 PM
valhallasw raised the priority of this task from to Needs Triage.
valhallasw updated the task description. (Show Details)
valhallasw added a project: Toolforge.
valhallasw subscribed.
Restricted Application added a project: Cloud-Services. · View Herald TranscriptDec 3 2015, 3:34 PM
Restricted Application added subscribers: StudiesWorld, Aklapper. · View Herald Transcript
valhallasw mentioned this in T120210: tools-mail: check SPF of sender before forwarding email.Dec 3 2015, 3:34 PM
valhallasw triaged this task as Low priority.Dec 3 2015, 3:36 PM
valhallasw added a project: Mail.Dec 22 2015, 9:32 PM
scfc mentioned this in T116806: phabricator.wikimedia.org has no SPF record.Apr 12 2016, 11:18 PM
scfc subscribed.Apr 12 2016, 11:24 PM

The name for this is "Sender Rewriting Scheme" (SRS); one solution is at https://github.com/Exim/exim/wiki/SRS. I think it should also be possible with pure exim, but I haven't looked deeper into this.

valhallasw moved this task from Backlog to Ready to be worked on on the Toolforge board.May 27 2016, 12:23 PM
• bd808 mentioned this in T249114: E-mails from noreply@pypi.org to tools.pywikibot@tools.wmflabs.org are not forwarded to certain recipients due to SPF.Apr 1 2020, 9:23 PM
• bd808 mentioned this in T249237: Fix Cloud VPS and Toolforge mail servers to work with the modern internet.Apr 2 2020, 4:36 PM
• bd808 added a parent task: T249237: Fix Cloud VPS and Toolforge mail servers to work with the modern internet.Apr 2 2020, 7:54 PM
herron added subscribers: aborrero, herron.May 4 2020, 10:44 PM

On paper we should be able to adapt an exim SRS config like this example https://github.com/hn/exim-misc/blob/master/exim-srs-sender-rewriting-scheme.conf to work with the tools.wmflabs.org forwarding aliases.

If that sounds workable I think the next step would be to stand up an environment and domain that mimmicks tools.wmflabs.org and prototype/validate this approach. If so, do you have a place/project in mind to get started on it @aborrero ?

aborrero renamed this task from correctly envelope forwarded email to Toolforge: correctly envelope forwarded email.May 5 2020, 9:51 AM
aborrero removed a project: Cloud-Services.
aborrero added a comment.May 5 2020, 9:56 AM

We usually use the toolsbeta project as the testing environment for tools (toolforge). That being said, I'm not sure if we would be able to test everything in toolsbeta (not sure if the email environment is the same or allows the same workflows that in tools). The domain there would be toolsbeta.wmflabs.org.

Other option we might consider is to directly do our tests with the toolforge.org domain. This new domain has no existing email users, so we can freely test in the tools project without risk of breaking current use cases.
Eventually we would like to provide email from this domain, so it might be a good idea to start using it anyway.

Perhaps both options are not exclusive. We can spin up the email setup in toolsbeta and then work on introducing toolforge.org for email once the setup is ready. Let's do this! I will create the VM now and give you access to it @herron

Stashbot added a comment.May 5 2020, 9:59 AM

Mentioned in SAL (#wikimedia-cloud) [2020-05-05T09:59:01Z] <arturo> created VM toolsbeta-mail-01 (T120225)

Stashbot added a comment.May 5 2020, 10:04 AM

Mentioned in SAL (#wikimedia-cloud) [2020-05-05T10:04:09Z] <arturo> add herron as user and projectadmin, we will work on the email setup (T120225)

Stashbot added a comment.May 8 2020, 12:24 PM

Mentioned in SAL (#wikimedia-cloud) [2020-05-08T12:24:07Z] <arturo> added puppet prefix toolsbeta-email (T120225)

aborrero added a comment.May 8 2020, 12:44 PM

I tried setting up the testing server. There is something going on with letsencrypt:

⌂66% aborrero@toolsbeta-mail-01:~ 1m49s $ sudo run-puppet-agent
Info: Using configured environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Retrieving locales
Info: Loading facts
Info: Caching catalog for toolsbeta-mail-01.toolsbeta.eqiad.wmflabs
Info: Applying configuration version '(100a62ce82) Alexandros Kosiaris - redis: Restore $slaveof parameter'
Notice: The LDAP client stack for this host is: sssd/sudo
Notice: /Stage[main]/Profile::Ldap::Client::Labs/Notify[LDAP client stack]/message: defined 'message' as 'The LDAP client stack for this host is: sssd/sudo'
Notice: /Stage[main]/Profile::Toolforge::Mailrelay/Letsencrypt::Cert::Integrated[tools_mail]/Exec[acme-setup-acme-tools_mail]/returns: Getting ACME cert /etc/acme/cert/tools_mail.crt
Notice: /Stage[main]/Profile::Toolforge::Mailrelay/Letsencrypt::Cert::Integrated[tools_mail]/Exec[acme-setup-acme-tools_mail]/returns: Traceback (most recent call last):
Notice: /Stage[main]/Profile::Toolforge::Mailrelay/Letsencrypt::Cert::Integrated[tools_mail]/Exec[acme-setup-acme-tools_mail]/returns:   File "/usr/local/sbin/acme-setup", line 508, in <module>
Notice: /Stage[main]/Profile::Toolforge::Mailrelay/Letsencrypt::Cert::Integrated[tools_mail]/Exec[acme-setup-acme-tools_mail]/returns:     main()
Notice: /Stage[main]/Profile::Toolforge::Mailrelay/Letsencrypt::Cert::Integrated[tools_mail]/Exec[acme-setup-acme-tools_mail]/returns:   File "/usr/local/sbin/acme-setup", line 504, in main
Notice: /Stage[main]/Profile::Toolforge::Mailrelay/Letsencrypt::Cert::Integrated[tools_mail]/Exec[acme-setup-acme-tools_mail]/returns:     key_uid, key_gid)
Notice: /Stage[main]/Profile::Toolforge::Mailrelay/Letsencrypt::Cert::Integrated[tools_mail]/Exec[acme-setup-acme-tools_mail]/returns:   File "/usr/local/sbin/acme-setup", line 477, in acme_setup
Notice: /Stage[main]/Profile::Toolforge::Mailrelay/Letsencrypt::Cert::Integrated[tools_mail]/Exec[acme-setup-acme-tools_mail]/returns:     exp_rand, chal_dir, acme_user, svc, force_crt)
Notice: /Stage[main]/Profile::Toolforge::Mailrelay/Letsencrypt::Cert::Integrated[tools_mail]/Exec[acme-setup-acme-tools_mail]/returns:   File "/usr/local/sbin/acme-setup", line 402, in ensure_crt_acme
Notice: /Stage[main]/Profile::Toolforge::Mailrelay/Letsencrypt::Cert::Integrated[tools_mail]/Exec[acme-setup-acme-tools_mail]/returns:     ensure_real_fs(tls_crt, 0o644, 0, 0, False, cert_create, cert_force)
Notice: /Stage[main]/Profile::Toolforge::Mailrelay/Letsencrypt::Cert::Integrated[tools_mail]/Exec[acme-setup-acme-tools_mail]/returns:   File "/usr/local/sbin/acme-setup", line 205, in ensure_real_fs
Notice: /Stage[main]/Profile::Toolforge::Mailrelay/Letsencrypt::Cert::Integrated[tools_mail]/Exec[acme-setup-acme-tools_mail]/returns:     creator()
Notice: /Stage[main]/Profile::Toolforge::Mailrelay/Letsencrypt::Cert::Integrated[tools_mail]/Exec[acme-setup-acme-tools_mail]/returns:   File "/usr/local/sbin/acme-setup", line 394, in cert_create
Notice: /Stage[main]/Profile::Toolforge::Mailrelay/Letsencrypt::Cert::Integrated[tools_mail]/Exec[acme-setup-acme-tools_mail]/returns:     acme_challenge(id, cert_dir, acct_key, csr, chal_dir, acme_user)
Notice: /Stage[main]/Profile::Toolforge::Mailrelay/Letsencrypt::Cert::Integrated[tools_mail]/Exec[acme-setup-acme-tools_mail]/returns:   File "/usr/local/sbin/acme-setup", line 369, in acme_challenge
Notice: /Stage[main]/Profile::Toolforge::Mailrelay/Letsencrypt::Cert::Integrated[tools_mail]/Exec[acme-setup-acme-tools_mail]/returns:     % (" ".join(args), p.returncode, p_err))
Notice: /Stage[main]/Profile::Toolforge::Mailrelay/Letsencrypt::Cert::Integrated[tools_mail]/Exec[acme-setup-acme-tools_mail]/returns: Exception: Command >>/usr/local/sbin/acme_tiny.py --account-key /etc/acme/acct/acct.key --csr /etc/acme/csr/tools_mail.pem --acme-dir /var/acme/challenge<< failed, exit code 1, stderr:
Notice: /Stage[main]/Profile::Toolforge::Mailrelay/Letsencrypt::Cert::Integrated[tools_mail]/Exec[acme-setup-acme-tools_mail]/returns: Parsing account key...
Notice: /Stage[main]/Profile::Toolforge::Mailrelay/Letsencrypt::Cert::Integrated[tools_mail]/Exec[acme-setup-acme-tools_mail]/returns: Parsing CSR...
Notice: /Stage[main]/Profile::Toolforge::Mailrelay/Letsencrypt::Cert::Integrated[tools_mail]/Exec[acme-setup-acme-tools_mail]/returns: Registering account...
Notice: /Stage[main]/Profile::Toolforge::Mailrelay/Letsencrypt::Cert::Integrated[tools_mail]/Exec[acme-setup-acme-tools_mail]/returns: Traceback (most recent call last):
Notice: /Stage[main]/Profile::Toolforge::Mailrelay/Letsencrypt::Cert::Integrated[tools_mail]/Exec[acme-setup-acme-tools_mail]/returns:   File "/usr/local/sbin/acme_tiny.py", line 234, in <module>
Notice: /Stage[main]/Profile::Toolforge::Mailrelay/Letsencrypt::Cert::Integrated[tools_mail]/Exec[acme-setup-acme-tools_mail]/returns:     main(sys.argv[1:])
Notice: /Stage[main]/Profile::Toolforge::Mailrelay/Letsencrypt::Cert::Integrated[tools_mail]/Exec[acme-setup-acme-tools_mail]/returns:   File "/usr/local/sbin/acme_tiny.py", line 230, in main
Notice: /Stage[main]/Profile::Toolforge::Mailrelay/Letsencrypt::Cert::Integrated[tools_mail]/Exec[acme-setup-acme-tools_mail]/returns:     signed_crt = get_crt(args.account_key, args.csr, args.acme_dir, log=LOGGER, CA=args.ca)
Notice: /Stage[main]/Profile::Toolforge::Mailrelay/Letsencrypt::Cert::Integrated[tools_mail]/Exec[acme-setup-acme-tools_mail]/returns:   File "/usr/local/sbin/acme_tiny.py", line 114, in get_crt
Notice: /Stage[main]/Profile::Toolforge::Mailrelay/Letsencrypt::Cert::Integrated[tools_mail]/Exec[acme-setup-acme-tools_mail]/returns:     raise ValueError("Error registering: {0} {1}".format(code, result))
Notice: /Stage[main]/Profile::Toolforge::Mailrelay/Letsencrypt::Cert::Integrated[tools_mail]/Exec[acme-setup-acme-tools_mail]/returns: ValueError: Error registering: 403 {
Notice: /Stage[main]/Profile::Toolforge::Mailrelay/Letsencrypt::Cert::Integrated[tools_mail]/Exec[acme-setup-acme-tools_mail]/returns:   "type": "urn:acme:error:unauthorized",
Notice: /Stage[main]/Profile::Toolforge::Mailrelay/Letsencrypt::Cert::Integrated[tools_mail]/Exec[acme-setup-acme-tools_mail]/returns:   "detail": "Account creation on ACMEv1 is disabled. Please upgrade your ACME client to a version that supports ACMEv2 / RFC 8555. See https://community.letsencrypt.org/t/end-of-life-plan-for-acmev1/88430 for details.",
Notice: /Stage[main]/Profile::Toolforge::Mailrelay/Letsencrypt::Cert::Integrated[tools_mail]/Exec[acme-setup-acme-tools_mail]/returns:   "status": 403
Notice: /Stage[main]/Profile::Toolforge::Mailrelay/Letsencrypt::Cert::Integrated[tools_mail]/Exec[acme-setup-acme-tools_mail]/returns: }
Error: '/usr/local/sbin/acme-setup -i tools_mail -s mail.toolsbeta.wmflabs.org --key-user root --key-group Debian-exim -m acme -w nginx' returned 1 instead of one of [0]
Error: /Stage[main]/Profile::Toolforge::Mailrelay/Letsencrypt::Cert::Integrated[tools_mail]/Exec[acme-setup-acme-tools_mail]/returns: change from 'notrun' to ['0'] failed: '/usr/local/sbin/acme-setup -i tools_mail -s mail.toolsbeta.wmflabs.org --key-user root --key-group Debian-exim -m acme -w nginx' returned 1 instead of one of [0]
Info: Stage[main]: Unscheduling all events on Stage[main]
Notice: Applied catalog in 14.19 seconds

I guess we don't have the rest of the setup to generate certs for this domain.

Stashbot added a comment.May 8 2020, 12:48 PM

Mentioned in SAL (#wikimedia-cloud) [2020-05-08T12:48:40Z] <arturo> allocated floating IP 185.15.56.12 for the VM toolsbeta-email-01 and FQDN mail.toolsbeta.wmflabs.org (T120225)

Krenair mentioned this in T252199: Stop using letsencrypt::cert::integrated.May 8 2020, 1:07 PM
Krenair subscribed.
aborrero added a comment.May 14 2020, 12:02 PM

I just opened T252762: tools/toolsbeta: improve acme-chief integration which is blocking a correct deployment of the mail server in toolsbeta for testing.

gerritbot added a comment.May 19 2020, 10:53 AM

Change 597257 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] toolforge: mailrelay: introduce support for disabling TLS

https://gerrit.wikimedia.org/r/597257

gerritbot added a project: Patch-For-Review.May 19 2020, 10:53 AM
gerritbot added a comment.May 19 2020, 11:04 AM

Change 597257 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] toolforge: mailrelay: introduce support for disabling TLS

https://gerrit.wikimedia.org/r/597257

gerritbot added a comment.Jun 23 2020, 9:33 AM

Change 607251 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] toolforge: mailrelay: migrate TLS cert to acme-chief

https://gerrit.wikimedia.org/r/607251

gerritbot added a comment.Jun 23 2020, 12:45 PM

Change 607279 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] toolforge: mailrelay: introduce SRS to correctly envelope forwarded emails

https://gerrit.wikimedia.org/r/607279

gerritbot added a comment.Jun 23 2020, 4:01 PM

Change 607251 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] toolforge: mailrelay: migrate TLS cert to acme-chief

https://gerrit.wikimedia.org/r/607251

Stashbot added a comment.Jun 24 2020, 11:24 AM

Mentioned in SAL (#wikimedia-cloud) [2020-06-24T11:24:00Z] <arturo> fix MX record for toolsbeta.wmflabs.org (missing trailing dot) T120225

Stashbot added a comment.Jun 24 2020, 11:26 AM

Mentioned in SAL (#wikimedia-cloud) [2020-06-24T11:26:05Z] <arturo> add TXT record "v=spf1 mx -all" T120225

aborrero raised the priority of this task from Low to High.Jun 24 2020, 11:26 AM
aborrero added a project: cloud-services-team (Kanban).
aborrero moved this task from Inbox to Doing on the cloud-services-team (Kanban) board.
Stashbot added a comment.Jun 24 2020, 12:55 PM

Mentioned in SAL (#wikimedia-cloud) [2020-06-24T12:55:18Z] <arturo> live-hacking puppetmaster with https://gerrit.wikimedia.org/r/c/operations/puppet/+/607279 (T120225)

gerritbot added a comment.Jun 25 2020, 12:59 PM

Change 607279 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] toolforge: mailrelay: introduce SRS to correctly envelope forwarded emails

https://gerrit.wikimedia.org/r/607279

aborrero closed this task as Resolved.Jun 25 2020, 1:10 PM
aborrero claimed this task.

I think this is fixed now.

If you explore the email headers you should see stuff like this:

Return-Path: <SRS0=706ec=4279=protonmail.com=arturobg@toolsbeta.wmflabs.org>
Received: from mail.toolsbeta.wmflabs.org (mail.toolsbeta.wmflabs.org. [185.15.56.12])
        by mx.google.com with ESMTPS id v7si11960456qkv.74.2020.06.25.05.47.56
        for <arturo.borrero.glez@gmail.com>
        (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);
        Thu, 25 Jun 2020 05:47:56 -0700 (PDT)
Received-SPF: pass (google.com: domain of srs0=706ec=4279=protonmail.com=arturobg@toolsbeta.wmflabs.org designates 185.15.56.12 as permitted sender) client-ip=185.15.56.12;
X-SRS-Rewrite: SMTP return-path rewritten from <arturobg@protonmail.com> by mail.toolsbeta.wmflabs.org

Note the SRS0= and srs0= data that was injected by the config we have now. Also, the new X-SRS-Rewrite control header is a symptom the system is working as expected.

I'm closing the task now, please feel free to reopen if required.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits