Pinging again as July is almost upon us, which is when we hope to run the wikimedia.org vs wikipedia.org domain test!

We plan to send the email on 7/20, so it would be ideal to have all DNS changes in place by 7/15. Can we make this happen?

This ticket follows up on T94052 where the idea of A:B testing was raised.

faidon, dpatrick: do either of you have any concerns about the proposal to A:B test use of the wikipedia.org domain for sending donor campaigin email as proposed?

Since this allows us to gather the data mentioned at T94052#1171831 and elsewhere in that ticket, and it provides FR with information that may increase the efficacy of their work, I'm okay with moving forward with this, with some caveats:

• We do not also need to allow Silverpop to host a website under wikipedia.org for click tracking, etc.;
• We use the granularity limiter ("g="), and set it to the specific mailbox which is originating e-mail;
• We have a statement from Silverpop describing the methods they use for protection of signing keys, and detection of key compromise and notification of customers in the event of key compromise; and
• We limit the amount of time the TXT record is in DNS.

The only requirement from my side would be to use a granularity limiter ("g=donate" or "g=donate*" perhaps?), which I'm guessing should be a fairly trivial addition. While you're at it, please add one for wikimedia.org — this is vulnerability that we've had for some time now that it's about time we close.

Other than that: please be careful when adding the SPF record; I believe wikipedia.org doesn't have any, so you'd need to add our own before adding Silverpop's, otherwise all of our Wiki emails will suddenly fail SPF. A better idea would be to perhaps just add an include:wikimedia.org SPF record instead of Silverpop's ip4.

Thanks @dpatrick & @faidon.

@CCogdill_WMF so the next step--could you get a statement from Silverpop to the effect of what dpatrick describes above? Also, what wikipedia.org address(es) do you plan to use as From- and envelope-senders?

Change 298484 had a related patch set uploaded (by Jgreen):
Remove gratuitous donate.wiki[mp]edia.org SPF records.

https://gerrit.wikimedia.org/r/298484

Change 298492 had a related patch set uploaded (by Faidon Liambotis):
Revert "spf records for wikipedia.org, see T135410"

https://gerrit.wikimedia.org/r/298492

Change 298492 merged by Faidon Liambotis:
Revert "spf records for wikipedia.org, see T135410"

https://gerrit.wikimedia.org/r/298492

Change 298500 had a related patch set uploaded (by Jgreen):
SPF record for wikipedia.org and domains sharing that zonefile.

https://gerrit.wikimedia.org/r/298500

Change 298484 abandoned by Jgreen:
Remove gratuitous donate.wiki[mp]edia.org SPF records.

Reason:
part of cascade of errors, starting over

https://gerrit.wikimedia.org/r/298484

Thanks @faidon and @dpatrick for making this possible, and in the nick of time! I really appreciate it.

I have a few notes/follow up questions for you, regarding @dpatrick's list:

• We do not also need to allow Silverpop to host a website under wikipedia.org for click tracking, etc.

IBM/Silverpop no longer does any clicktracking under our domains and the DNS to add click tracking has been removed, so there would be no way that would be possible.

• We use the granularity limiter ("g="), and set it to the specific mailbox which is originating e-mail

In IBM's system, the sender is simply the from email address, so usually that will just be i=donate@wikipedia.org. Sometimes we use a different from address, and the i= value would change. Is there a way to do this that doesn't restrict us from changing whatever comes before *@wikimedia.org?

• We have a statement from Silverpop describing the methods they use for protection of signing keys, and detection of key compromise and notification of customers in the event of key compromise.

We're not confident they will provide this, as their stance is often that telling someone how they protect their data in itself is a security issue.... Will let you know.

• We limit the amount of time the TXT record is in DNS.

Sure, though we would like the room to reconsider adding it as a semi-permanent part of DNS if this wins.

To follow up on my last comment, I created T140316 which addresses the granularity limiter for wikimedia.org. Can we discuss wikimedia.org separately, and just focus on wikipedia.org for this test?

We can set the wikipedia.org record so g=donate@wikimedia.org.

I'm waiting to hear back if IBM will provide security specs on key handling. Is there anything else you need from me?

Distilling the discussion to a proposed config, here's the DKIM record I think we would add to the wikipedia.org zone file for this test:

spop1024._domainkey 1H IN TXT "v=DKIM1; k=rsa; g=donate; p=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDpnujkAKfZWUN6W07scekor+i8wGvRGq3Ns9bk3nmgWovfbBebFwzHoNv3Oaq9VMf1cdTf53fuQY5NVZQ+hGZXqJ4mkcNJnSC0BZuGC45kUWjyOhB+Pxf0o9AuGM8QCQwm4nnvNqUe/lME7bFTzrtCSrF7sG0UxvwM2AeDEZj5NQIDAQAB"

Note that one implication of adding this record is that it will apply to several other domains as well, because of the way we template our DNS zones.

In T135410#2462453, @Jgreen wrote:

Distilling the discussion to a proposed config, here's the DKIM record I think we would add to the wikipedia.org zone file for this test:

spop1024._domainkey 1H IN TXT "v=DKIM1; k=rsa; g=donate; p=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDpnujkAKfZWUN6W07scekor+i8wGvRGq3Ns9bk3nmgWovfbBebFwzHoNv3Oaq9VMf1cdTf53fuQY5NVZQ+hGZXqJ4mkcNJnSC0BZuGC45kUWjyOhB+Pxf0o9AuGM8QCQwm4nnvNqUe/lME7bFTzrtCSrF7sG0UxvwM2AeDEZj5NQIDAQAB"

Note that one implication of adding this record is that it will apply to several other domains as well, because of the way we template our DNS zones.

This looks good to me.

I think all the necessary DNS config changes are in https://gerrit.wikimedia.org/r/#/c/298500/ and that is waiting for review by Ops folks. I'm not sure whether that will be approved until we hear back from IBM on security specs.

Once the DNS change goes through, please test before running a real mailing and also remember that it's possible that DNS caching can delay the change from being seen by some mailservers.

Change 298500 merged by Jgreen:
SPF/DKIM records for wikipedia.org and domains sharing that zonefile.

https://gerrit.wikimedia.org/r/298500

DNS changes are deployed, let's leave this ticket open until the fundraising mailing tests are complete and the DKIM record has been removed.

Thanks @Jgreen, everything looks good on our end.

We're still pushing IBM for security specs. We at least have some
information on how their server's mailers use and save DKIM, but are seeing
what more we can find.

I hope to send this email out on Tuesday, 7/26. I'll update again at the
end of this week.

After pushing IBM for a couple weeks, they finally sent us this response today:

“After reviewing with the IBM team, unfortunately this question is not something that we would be able to assist you with. Given the Information sensitivity, this would be classed as proprietary information.”

We're not sure what other tactic to take. Considering we have been able to send safely for the past 5 years, can we go ahead with the test without having this extra information?

IBM tried to validate wikipedia.org, but the validation failed because the the p= value was wrong. The correct key is:

MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDTSe5iLtjAxBaC7dh0isO18kLQiVM8XSa/X0amoJvULa2o9mmV01UesAqGqBK+yPza2B2yOGyOTrDRBkPfbGek9MoVVXIkuezbG6DFz2ytWg581nSlOewA+QMe+RRYWaYlwmH1NHVpUUO2L2/waFsWLNOSMNyn5PWjqBKBBmffVwIDAQAB

But what is live is:

MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDpnujkAKfZWUN6W07scekor+i8wGvRGq3Ns9bk3nmgWovfbBebFwzHoNv3Oaq9VMf1cdTf53fuQY5NVZQ+hGZXqJ4mkcNJnSC0BZuGC45kUWjyOhB+Pxf0o9AuGM8QCQwm4nnvNqUe/lME7bFTzrtCSrF7sG0UxvwM2AeDEZj5NQIDAQAB

Can we update that?

In T135410#2487449, @CCogdill_WMF wrote:

IBM tried to validate wikipedia.org, but the validation failed because the the p= value was wrong. The correct key is:

MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDTSe5iLtjAxBaC7dh0isO18kLQiVM8XSa/X0amoJvULa2o9mmV01UesAqGqBK+yPza2B2yOGyOTrDRBkPfbGek9MoVVXIkuezbG6DFz2ytWg581nSlOewA+QMe+RRYWaYlwmH1NHVpUUO2L2/waFsWLNOSMNyn5PWjqBKBBmffVwIDAQAB

But what is live is:

MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDpnujkAKfZWUN6W07scekor+i8wGvRGq3Ns9bk3nmgWovfbBebFwzHoNv3Oaq9VMf1cdTf53fuQY5NVZQ+hGZXqJ4mkcNJnSC0BZuGC45kUWjyOhB+Pxf0o9AuGM8QCQwm4nnvNqUe/lME7bFTzrtCSrF7sG0UxvwM2AeDEZj5NQIDAQAB

Can we update that?

Whoops. That's the spop key from wikimedia.org, fixing...

... and the fix is deployed.

Change 300565 had a related patch set uploaded (by Jgreen):
corrected public key for spop1024._domainkey in wikipedia.org zone

https://gerrit.wikimedia.org/r/300565

Change 300565 merged by Jgreen:
corrected public key for spop1024._domainkey in wikipedia.org zone

https://gerrit.wikimedia.org/r/300565

Thanks, Jeff!

Confirmed with IBM that the updated key works, and we've done some validation — looks like DKIM and DMARC pass, so we should be good to send.

I'm going to schedule this to go out to 215k US donors tomorrow morning. I'll confirm when it goes out and we can delete the record at that point. If you have any concerns in the meantime, please let me know ASAP! Thanks so much.

In T135410#2493042, @CCogdill_WMF wrote:

Confirmed with IBM that the updated key works, and we've done some validation — looks like DKIM and DMARC pass, so we should be good to send.

I'm going to schedule this to go out to 215k US donors tomorrow morning. I'll confirm when it goes out and we can delete the record at that point. If you have any concerns in the meantime, please let me know ASAP! Thanks so much.

@CCogdill_WMF How did this go?

@Jgreen Have the DNS records been removed?

DNS records have not been removed yet, I haven't heard whether they're
done with the A/B tests.

We can remove the DNS record, the test is done and the tail of email opens should be about done. Thank you!

We are not seeing a significant difference in the performance between the two domains. wikipedia.org does seem to get less abuse complaints, but we get so few as it is that we can't call the difference significant.

The one thing I want to note is I predict I'll ask if we can change our spf record to ~all at some point in the future. The woman considered one of the best email deliverability experts in the field says:

My one rule for SPF is never use ?all. Just. No. In the spec, ?all is “testing” mode... Unless they really are testing, but even then you shouldn’t see ?all on records for weeks or months.

From https://wordtothewise.com/2016/07/spf-all/

This is almost definitely impacting us with AOL, but we predict eventually it will hurt us on other mail clients. So if one of these two domains -- wikipedia.org or wikimedia.org -- is better suited to ~all, I'd like to keep this in mind.

Thanks again for letting me run this test.

@patrick before the test wikipedia.org had no SPF record, and current
record tracks wikimedia.org:

"v=spf1 include:wikimedia.org ?all"

We probably want a policy of some kind going forward, should we stick with
this version?

DKIM is easy, I will commit a change to remove that.

In T135410#2522057, @Jgreen wrote:

"v=spf1 include:wikimedia.org ?all"

We probably want a policy of some kind going forward, should we stick with > this version?

Yeah, that's fine.

Can we add the g=donate to wikimedia.org now as well? Shall I open another ticket about this?

Change 303159 had a related patch set uploaded (by Jgreen):
remove temporary fundraising DKIM record from wikipedia.org zone template

https://gerrit.wikimedia.org/r/303159

In T135410#2522087, @faidon wrote:

In T135410#2522057, @Jgreen wrote:

"v=spf1 include:wikimedia.org ?all"

We probably want a policy of some kind going forward, should we stick with > this version?

Yeah, that's fine.

Can we add the g=donate to wikimedia.org now as well? Shall I open another ticket about this?

Yeah, better as a separate task. I opened T142205.

Change 303159 merged by Faidon Liambotis:
remove temporary fundraising DKIM record from wikipedia.org zone template

https://gerrit.wikimedia.org/r/303159