@BBlack - any updates?

Hi Anusha, Greg,

Looking into this. Unfortunately it seems the way this is being implemented, we would effectively be signing away complete control of our email security settings/policy for wikimedia.org to a set of third party companies. I'm afraid that's rather problematic, because specifically wikimedia.org is one of our project domains, and our main infrastructure domain and as our primary corporate email domain, and a lot of things depend on the (email) security of that domain. This would allow these third parties to set arbitrary security policies for that domain, and for example allow for any of our email addresses or staff members to be impersonated (which is actually what seems to be desired here, yet in a very restricted way).

Because (unfortunately) this is the primary domain historically used for all corporate mail as well as infrastructure, a lot of security sensitive procedures depend on the email security of wikimedia.org, e.g. generation of TLS certificates and other aspects that protect the privacy and security of both our global users and our staff/organization. Therefore handing over full control of this for domain wikimedia.org to third parties is high risk. As it is proposed now, through either a malicious attempt (e.g. a compromise) or even an unintentional configuration mistake it may be possible for anyone to gain control of it and impersonate @wikimedia.org addresses, which could have grave consequences. And we would have not even have visibility into it when that happens, no control over it, and may never realize it happened.

Would it perhaps be possible to find an alternative or compromise to (mostly) achieve the benefits you're looking for without requiring this highly risky implementation using wikimedia.org? For example, without being exhaustive or having deeply researched these, right now I can see the following alternatives:

• Would it be possible to use another email domain than wikimedia.org; either a subdomain (e.g. @comms.wikimedia.org or anything else you pick that is not yet in use) or another domain altogether (perhaps @wikimediafoundation.org is suitable)? We should be able to make those email domains work as aliases for specific staff members and make them look entirely legit that way.

• Or, would it be possible to have the required DomainKey (DKIM) records installed by us in our DNS instead of CNAMEd to them, and also limited/scoped to only a restricted set of e-mail addresses? For example, we could authorize MuckRack to send out emails for only some specific @wikimedia.org addresses, e.g. some Communications team members, and no others. This would however require support/cooperation from MuckRack.

• Or, would it be possible for MuckRack to send out emails via our corporate email relays instead of their own / AmazonSES perhaps? This would also alleviate the issue, but again would require their support/cooperation.

I'm sorry to have to make this difficult, but specifically that domain really limits what we can do in a many ways. Hopefully in the future we are able to split these functions (corporate email, wiki project domains, and infrastructure domain) over multiple disjunct domains, so we can be much more flexible and lower our risks altogether.

Thanks,

Mark

@mark - Thank you very much for that thoughtful and helpful reply!

Talking it over, we would like to try the first option if you believe that will work.

So how do we go about getting this setup?

Anusha Alikhan
aalikhan@pr.wikimedia.org

Samantha Lien
slien@pr.wikimedia.org

From MuckRack:
TXT pr.wikimedia.org
amazonses:QeZasSSTVw5sDiCQmdzG4z4UuLgkRtceplXtv2SI2BY=

CNAME lecokkzzn6akfzusmban7ufr6xw2g5ye._domainkey.pr.wikimedia.org lecokkzzn6akfzusmban7ufr6xw2g5ye.dkim.amazonses.com

CNAME kilujm6hbzryz5wrq5l3dkb5la5rrxgc._domainkey.pr.wikimedia.org kilujm6hbzryz5wrq5l3dkb5la5rrxgc.dkim.amazonses.com

CNAME 4njdxtpft2kiksyew6qbqcxnxip4cjer._domainkey.pr.wikimedia.org 4njdxtpft2kiksyew6qbqcxnxip4cjer.dkim.amazonses.com

Hi @mark - Just checking in on the above. Thanks.

In T231387#5471833, @Varnent wrote:

@mark - Thank you very much for that thoughtful and helpful reply!

Talking it over, we would like to try the first option if you believe that will work.

So how do we go about getting this setup?

Anusha Alikhan
aalikhan@pr.wikimedia.org

Samantha Lien
slien@pr.wikimedia.org

I believe that will work, yes.

@herron Could you please take this? See the above; implement pr.wikimedia.org as a mail-only domain with the above aliases in Exim for a few communications team members, aliasing to their normal (gmail) addresses. Then once that works, install the above CNAME records for pr.wikimedia.org DKIM to MuckRack so they are allowed to send mail for that domain only. Thanks!

Change 536313 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] exim: add pr.wikimedia.org to wikimedia_domains

https://gerrit.wikimedia.org/r/536313

Change 536318 had a related patch set uploaded (by Herron; owner: Herron):
[operations/dns@master] dns: add mail records for pr.wikimedia.org

https://gerrit.wikimedia.org/r/536318

@mark you bet! I've uploaded a few patches to get mail flowing for this subdomain.

@Varnent were there any SPF records provided by muckrack/ses? Also could you please verify that the below aliases look good? To the left is the user-part for @pr.wikimedia.org, and on the right is the destination.

aalikhan: aalikhan@wikimedia.org
slien:    slien@wikimedia.org

Change 536318 merged by Herron:
[operations/dns@master] dns: add mail records for pr.wikimedia.org

https://gerrit.wikimedia.org/r/536318

Change 536313 merged by Herron:
[operations/puppet@production] exim: add pr.wikimedia.org to wikimedia_domains

https://gerrit.wikimedia.org/r/536313

What's the status of this? Is this done and working?

Aliases have been added, dns records are live, and test mail to myself works. Please see https://phabricator.wikimedia.org/T231387#5488828 for currently open follow-up questions.

Thank you! I tested and it's working.

From MuckRack:

do you know when the DNS records were added? We don't see any of them returning as valid on our side just yet.

Possible to share a screen shot of how those records were added so our team can provide feedback if needed?

DNS records were added Monday at approx. 10:30am Eastern.

Here is what I'm seeing in terms of responses from live DNS:

$ host -t MX pr.wikimedia.org
pr.wikimedia.org mail is handled by 10 mx1001.wikimedia.org.
pr.wikimedia.org mail is handled by 50 mx2001.wikimedia.org.

$ host -t TXT pr.wikimedia.org
pr.wikimedia.org descriptive text "amazonses:QeZasSSTVw5sDiCQmdzG4z4UuLgkRtceplXtv2SI2BY="

$ host lecokkzzn6akfzusmban7ufr6xw2g5ye._domainkey.pr.wikimedia.org
lecokkzzn6akfzusmban7ufr6xw2g5ye._domainkey.pr.wikimedia.org is an alias for lecokkzzn6akfzusmban7ufr6xw2g5ye.dkim.amazonses.com.

$ host kilujm6hbzryz5wrq5l3dkb5la5rrxgc._domainkey.pr.wikimedia.org
kilujm6hbzryz5wrq5l3dkb5la5rrxgc._domainkey.pr.wikimedia.org is an alias for kilujm6hbzryz5wrq5l3dkb5la5rrxgc.dkim.amazonses.com.

$ host 4njdxtpft2kiksyew6qbqcxnxip4cjer._domainkey.pr.wikimedia.org
4njdxtpft2kiksyew6qbqcxnxip4cjer._domainkey.pr.wikimedia.org is an alias for 4njdxtpft2kiksyew6qbqcxnxip4cjer.dkim.amazonses.com.

@herron : There's now two log entries in the /var/log/exim4/paniclog on mx1001.wikimedia.org related to DKIM, not sure whether we need to tweak something still or whether that's harmless noise (if it's the latter, it should ideally not spam paniclog):

2019-09-19 00:30:06 1iAkKj-00026a-WE DKIM: signing failed: RSA_LONG_LINE
2019-09-19 02:27:31 1iAmAM-0006eB-NK DKIM: signing failed: RSA_LONG_LINE