Page MenuHomePhabricator

Toolforge: correctly envelope forwarded email
Closed, ResolvedPublic

Description

The current settings cause the SPF record of the original from: host to be checked, rather than those of tools.wmflabs.org:

valhallasw@gmail.com -> valhallasw@tools.wmflabs.org -> valhallasw@arctus.nl

Delivered-To: valhallasw@arctus.nl
Received: by 10.107.55.7 with SMTP id e7csp52297ioa;
        Thu, 3 Dec 2015 07:29:28 -0800 (PST)
X-Received: by 10.140.232.65 with SMTP id d62mr12365942qhc.27.1449156568542;
        Thu, 03 Dec 2015 07:29:28 -0800 (PST)
Return-Path: <valhallasw@gmail.com>
Received: from mail.tools.wmflabs.org (mail.tools.wmflabs.org. [208.80.155.162])
        by mx.google.com with ESMTPS id w104si1741458qge.65.2015.12.03.07.29.28
        for <valhallasw@arctus.nl>
        (version=TLS1 cipher=AES128-SHA bits=128/128);
        Thu, 03 Dec 2015 07:29:28 -0800 (PST)
Received-SPF: softfail (google.com: domain of transitioning valhallasw@gmail.com does not designate 208.80.155.162 as permitted sender) client-ip=208.80.155.162;
Authentication-Results: mx.google.com;
       spf=softfail (google.com: domain of transitioning valhallasw@gmail.com does not designate 208.80.155.162 as permitted sender) smtp.mailfrom=valhallasw@gmail.com;
       dkim=pass header.i=@gmail.com;
       dmarc=pass (p=NONE dis=NONE) header.from=gmail.com
Received: from mail-ig0-f174.google.com ([209.85.213.174])
	by mail.tools.wmflabs.org with esmtp (Exim 4.76)
	(envelope-from <valhallasw@gmail.com>)
	id 1a4Vp2-0001sC-6p
	for valhallasw@tools.wmflabs.org; Thu, 03 Dec 2015 15:29:28 +0000
Received: by igbxm8 with SMTP id xm8so14673378igb.1
        for <valhallasw@tools.wmflabs.org>; Thu, 03 Dec 2015 07:29:20 -0800 (PST)

Event Timeline

valhallasw raised the priority of this task from to Needs Triage.
valhallasw updated the task description. (Show Details)
valhallasw added a project: Toolforge.
valhallasw added a subscriber: valhallasw.
Restricted Application added subscribers: StudiesWorld, Aklapper. · View Herald Transcript

The name for this is "Sender Rewriting Scheme" (SRS); one solution is at https://github.com/Exim/exim/wiki/SRS. I think it should also be possible with pure exim, but I haven't looked deeper into this.

On paper we should be able to adapt an exim SRS config like this example https://github.com/hn/exim-misc/blob/master/exim-srs-sender-rewriting-scheme.conf to work with the tools.wmflabs.org forwarding aliases.

If that sounds workable I think the next step would be to stand up an environment and domain that mimmicks tools.wmflabs.org and prototype/validate this approach. If so, do you have a place/project in mind to get started on it @aborrero ?

aborrero renamed this task from correctly envelope forwarded email to Toolforge: correctly envelope forwarded email.May 5 2020, 9:51 AM
aborrero removed a project: Cloud-Services.

We usually use the toolsbeta project as the testing environment for tools (toolforge). That being said, I'm not sure if we would be able to test everything in toolsbeta (not sure if the email environment is the same or allows the same workflows that in tools). The domain there would be toolsbeta.wmflabs.org.

Other option we might consider is to directly do our tests with the toolforge.org domain. This new domain has no existing email users, so we can freely test in the tools project without risk of breaking current use cases.
Eventually we would like to provide email from this domain, so it might be a good idea to start using it anyway.

Perhaps both options are not exclusive. We can spin up the email setup in toolsbeta and then work on introducing toolforge.org for email once the setup is ready. Let's do this! I will create the VM now and give you access to it @herron

Mentioned in SAL (#wikimedia-cloud) [2020-05-05T09:59:01Z] <arturo> created VM toolsbeta-mail-01 (T120225)

Mentioned in SAL (#wikimedia-cloud) [2020-05-05T10:04:09Z] <arturo> add herron as user and projectadmin, we will work on the email setup (T120225)

Mentioned in SAL (#wikimedia-cloud) [2020-05-08T12:24:07Z] <arturo> added puppet prefix toolsbeta-email (T120225)

I tried setting up the testing server. There is something going on with letsencrypt:

⌂66% aborrero@toolsbeta-mail-01:~ 1m49s $ sudo run-puppet-agent
Info: Using configured environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Retrieving locales
Info: Loading facts
Info: Caching catalog for toolsbeta-mail-01.toolsbeta.eqiad.wmflabs
Info: Applying configuration version '(100a62ce82) Alexandros Kosiaris - redis: Restore $slaveof parameter'
Notice: The LDAP client stack for this host is: sssd/sudo
Notice: /Stage[main]/Profile::Ldap::Client::Labs/Notify[LDAP client stack]/message: defined 'message' as 'The LDAP client stack for this host is: sssd/sudo'
Notice: /Stage[main]/Profile::Toolforge::Mailrelay/Letsencrypt::Cert::Integrated[tools_mail]/Exec[acme-setup-acme-tools_mail]/returns: Getting ACME cert /etc/acme/cert/tools_mail.crt
Notice: /Stage[main]/Profile::Toolforge::Mailrelay/Letsencrypt::Cert::Integrated[tools_mail]/Exec[acme-setup-acme-tools_mail]/returns: Traceback (most recent call last):
Notice: /Stage[main]/Profile::Toolforge::Mailrelay/Letsencrypt::Cert::Integrated[tools_mail]/Exec[acme-setup-acme-tools_mail]/returns:   File "/usr/local/sbin/acme-setup", line 508, in <module>
Notice: /Stage[main]/Profile::Toolforge::Mailrelay/Letsencrypt::Cert::Integrated[tools_mail]/Exec[acme-setup-acme-tools_mail]/returns:     main()
Notice: /Stage[main]/Profile::Toolforge::Mailrelay/Letsencrypt::Cert::Integrated[tools_mail]/Exec[acme-setup-acme-tools_mail]/returns:   File "/usr/local/sbin/acme-setup", line 504, in main
Notice: /Stage[main]/Profile::Toolforge::Mailrelay/Letsencrypt::Cert::Integrated[tools_mail]/Exec[acme-setup-acme-tools_mail]/returns:     key_uid, key_gid)
Notice: /Stage[main]/Profile::Toolforge::Mailrelay/Letsencrypt::Cert::Integrated[tools_mail]/Exec[acme-setup-acme-tools_mail]/returns:   File "/usr/local/sbin/acme-setup", line 477, in acme_setup
Notice: /Stage[main]/Profile::Toolforge::Mailrelay/Letsencrypt::Cert::Integrated[tools_mail]/Exec[acme-setup-acme-tools_mail]/returns:     exp_rand, chal_dir, acme_user, svc, force_crt)
Notice: /Stage[main]/Profile::Toolforge::Mailrelay/Letsencrypt::Cert::Integrated[tools_mail]/Exec[acme-setup-acme-tools_mail]/returns:   File "/usr/local/sbin/acme-setup", line 402, in ensure_crt_acme
Notice: /Stage[main]/Profile::Toolforge::Mailrelay/Letsencrypt::Cert::Integrated[tools_mail]/Exec[acme-setup-acme-tools_mail]/returns:     ensure_real_fs(tls_crt, 0o644, 0, 0, False, cert_create, cert_force)
Notice: /Stage[main]/Profile::Toolforge::Mailrelay/Letsencrypt::Cert::Integrated[tools_mail]/Exec[acme-setup-acme-tools_mail]/returns:   File "/usr/local/sbin/acme-setup", line 205, in ensure_real_fs
Notice: /Stage[main]/Profile::Toolforge::Mailrelay/Letsencrypt::Cert::Integrated[tools_mail]/Exec[acme-setup-acme-tools_mail]/returns:     creator()
Notice: /Stage[main]/Profile::Toolforge::Mailrelay/Letsencrypt::Cert::Integrated[tools_mail]/Exec[acme-setup-acme-tools_mail]/returns:   File "/usr/local/sbin/acme-setup", line 394, in cert_create
Notice: /Stage[main]/Profile::Toolforge::Mailrelay/Letsencrypt::Cert::Integrated[tools_mail]/Exec[acme-setup-acme-tools_mail]/returns:     acme_challenge(id, cert_dir, acct_key, csr, chal_dir, acme_user)
Notice: /Stage[main]/Profile::Toolforge::Mailrelay/Letsencrypt::Cert::Integrated[tools_mail]/Exec[acme-setup-acme-tools_mail]/returns:   File "/usr/local/sbin/acme-setup", line 369, in acme_challenge
Notice: /Stage[main]/Profile::Toolforge::Mailrelay/Letsencrypt::Cert::Integrated[tools_mail]/Exec[acme-setup-acme-tools_mail]/returns:     % (" ".join(args), p.returncode, p_err))
Notice: /Stage[main]/Profile::Toolforge::Mailrelay/Letsencrypt::Cert::Integrated[tools_mail]/Exec[acme-setup-acme-tools_mail]/returns: Exception: Command >>/usr/local/sbin/acme_tiny.py --account-key /etc/acme/acct/acct.key --csr /etc/acme/csr/tools_mail.pem --acme-dir /var/acme/challenge<< failed, exit code 1, stderr:
Notice: /Stage[main]/Profile::Toolforge::Mailrelay/Letsencrypt::Cert::Integrated[tools_mail]/Exec[acme-setup-acme-tools_mail]/returns: Parsing account key...
Notice: /Stage[main]/Profile::Toolforge::Mailrelay/Letsencrypt::Cert::Integrated[tools_mail]/Exec[acme-setup-acme-tools_mail]/returns: Parsing CSR...
Notice: /Stage[main]/Profile::Toolforge::Mailrelay/Letsencrypt::Cert::Integrated[tools_mail]/Exec[acme-setup-acme-tools_mail]/returns: Registering account...
Notice: /Stage[main]/Profile::Toolforge::Mailrelay/Letsencrypt::Cert::Integrated[tools_mail]/Exec[acme-setup-acme-tools_mail]/returns: Traceback (most recent call last):
Notice: /Stage[main]/Profile::Toolforge::Mailrelay/Letsencrypt::Cert::Integrated[tools_mail]/Exec[acme-setup-acme-tools_mail]/returns:   File "/usr/local/sbin/acme_tiny.py", line 234, in <module>
Notice: /Stage[main]/Profile::Toolforge::Mailrelay/Letsencrypt::Cert::Integrated[tools_mail]/Exec[acme-setup-acme-tools_mail]/returns:     main(sys.argv[1:])
Notice: /Stage[main]/Profile::Toolforge::Mailrelay/Letsencrypt::Cert::Integrated[tools_mail]/Exec[acme-setup-acme-tools_mail]/returns:   File "/usr/local/sbin/acme_tiny.py", line 230, in main
Notice: /Stage[main]/Profile::Toolforge::Mailrelay/Letsencrypt::Cert::Integrated[tools_mail]/Exec[acme-setup-acme-tools_mail]/returns:     signed_crt = get_crt(args.account_key, args.csr, args.acme_dir, log=LOGGER, CA=args.ca)
Notice: /Stage[main]/Profile::Toolforge::Mailrelay/Letsencrypt::Cert::Integrated[tools_mail]/Exec[acme-setup-acme-tools_mail]/returns:   File "/usr/local/sbin/acme_tiny.py", line 114, in get_crt
Notice: /Stage[main]/Profile::Toolforge::Mailrelay/Letsencrypt::Cert::Integrated[tools_mail]/Exec[acme-setup-acme-tools_mail]/returns:     raise ValueError("Error registering: {0} {1}".format(code, result))
Notice: /Stage[main]/Profile::Toolforge::Mailrelay/Letsencrypt::Cert::Integrated[tools_mail]/Exec[acme-setup-acme-tools_mail]/returns: ValueError: Error registering: 403 {
Notice: /Stage[main]/Profile::Toolforge::Mailrelay/Letsencrypt::Cert::Integrated[tools_mail]/Exec[acme-setup-acme-tools_mail]/returns:   "type": "urn:acme:error:unauthorized",
Notice: /Stage[main]/Profile::Toolforge::Mailrelay/Letsencrypt::Cert::Integrated[tools_mail]/Exec[acme-setup-acme-tools_mail]/returns:   "detail": "Account creation on ACMEv1 is disabled. Please upgrade your ACME client to a version that supports ACMEv2 / RFC 8555. See https://community.letsencrypt.org/t/end-of-life-plan-for-acmev1/88430 for details.",
Notice: /Stage[main]/Profile::Toolforge::Mailrelay/Letsencrypt::Cert::Integrated[tools_mail]/Exec[acme-setup-acme-tools_mail]/returns:   "status": 403
Notice: /Stage[main]/Profile::Toolforge::Mailrelay/Letsencrypt::Cert::Integrated[tools_mail]/Exec[acme-setup-acme-tools_mail]/returns: }
Error: '/usr/local/sbin/acme-setup -i tools_mail -s mail.toolsbeta.wmflabs.org --key-user root --key-group Debian-exim -m acme -w nginx' returned 1 instead of one of [0]
Error: /Stage[main]/Profile::Toolforge::Mailrelay/Letsencrypt::Cert::Integrated[tools_mail]/Exec[acme-setup-acme-tools_mail]/returns: change from 'notrun' to ['0'] failed: '/usr/local/sbin/acme-setup -i tools_mail -s mail.toolsbeta.wmflabs.org --key-user root --key-group Debian-exim -m acme -w nginx' returned 1 instead of one of [0]
Info: Stage[main]: Unscheduling all events on Stage[main]
Notice: Applied catalog in 14.19 seconds

I guess we don't have the rest of the setup to generate certs for this domain.

Mentioned in SAL (#wikimedia-cloud) [2020-05-08T12:48:40Z] <arturo> allocated floating IP 185.15.56.12 for the VM toolsbeta-email-01 and FQDN mail.toolsbeta.wmflabs.org (T120225)

I just opened T252762: tools/toolsbeta: improve acme-chief integration which is blocking a correct deployment of the mail server in toolsbeta for testing.

Change 597257 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] toolforge: mailrelay: introduce support for disabling TLS

https://gerrit.wikimedia.org/r/597257

Change 597257 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] toolforge: mailrelay: introduce support for disabling TLS

https://gerrit.wikimedia.org/r/597257

Change 607251 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] toolforge: mailrelay: migrate TLS cert to acme-chief

https://gerrit.wikimedia.org/r/607251

Change 607279 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] toolforge: mailrelay: introduce SRS to correctly envelope forwarded emails

https://gerrit.wikimedia.org/r/607279

Change 607251 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] toolforge: mailrelay: migrate TLS cert to acme-chief

https://gerrit.wikimedia.org/r/607251

Mentioned in SAL (#wikimedia-cloud) [2020-06-24T11:24:00Z] <arturo> fix MX record for toolsbeta.wmflabs.org (missing trailing dot) T120225

Mentioned in SAL (#wikimedia-cloud) [2020-06-24T11:26:05Z] <arturo> add TXT record "v=spf1 mx -all" T120225

aborrero raised the priority of this task from Low to High.Jun 24 2020, 11:26 AM
aborrero moved this task from Inbox to Doing on the cloud-services-team (Kanban) board.

Change 607279 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] toolforge: mailrelay: introduce SRS to correctly envelope forwarded emails

https://gerrit.wikimedia.org/r/607279

aborrero claimed this task.

I think this is fixed now.

If you explore the email headers you should see stuff like this:

Return-Path: <SRS0=706ec=4279=protonmail.com=arturobg@toolsbeta.wmflabs.org>
Received: from mail.toolsbeta.wmflabs.org (mail.toolsbeta.wmflabs.org. [185.15.56.12])
        by mx.google.com with ESMTPS id v7si11960456qkv.74.2020.06.25.05.47.56
        for <arturo.borrero.glez@gmail.com>
        (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);
        Thu, 25 Jun 2020 05:47:56 -0700 (PDT)
Received-SPF: pass (google.com: domain of srs0=706ec=4279=protonmail.com=arturobg@toolsbeta.wmflabs.org designates 185.15.56.12 as permitted sender) client-ip=185.15.56.12;
X-SRS-Rewrite: SMTP return-path rewritten from <arturobg@protonmail.com> by mail.toolsbeta.wmflabs.org

Note the SRS0= and srs0= data that was injected by the config we have now. Also, the new X-SRS-Rewrite control header is a symptom the system is working as expected.

I'm closing the task now, please feel free to reopen if required.