Page MenuHomePhabricator

SSL cert for links.email.wikimedia.org
Open, MediumPublic

Description

Back in 2015, we turned off our email service provider's (IBM's Watson Campaign Monitor, we call Silverpop) clicktracking feature because the only way we were able to track clicks on a custom domain was over http. The Foundation was moving to adopt an HSTS policy and couldn't have unsecured domains out there. So we started using landing page hits on donate.wiki as an approximation for clicks. That has worked well enough, but has three limitations/pain points:

a) It means we have to get our email performance data from 3 different sources, rather than 2, which slows down the stats process.

b) We aren't able to track clicks to any link not hosted on donate.wikimedia.org

c) We don't have individual clickthrough events recorded in the ESP so it's harder to set up behavioral email programs

Recently, Silverpop changed their policy and are allowing clients to purchase SSL certs for clicktracking domains, so we'd like to switch back to using our ESP for clicktracking. We need to get Ops' approval to use a subdomain for clicktracking again so we can then have them purchase a cert. For now, we need to confirm this information is correct:

2-Character Country: US
State: California
Locality/City: San Francisco
Organization Name (Company Name): Wikimedia Foundation, Inc.
Organization Unit (example: Marketing, Sales, etc): Fundraising
Common Name: links.email.donate.wikimedia.org (this is the subdomain we used before)
Contact Email Address: [REDACTED]

Silverpop will then provide a CSR for us to purchase the cert with. It's recommended that we buy a 3-year cert. As to what level, that is up to Ops, but the user virtually never sees this site load so we don't think we need to spend much on it. Then we provide will then provide the cert to Trilogy and they will provide it to IBM. After that it takes just 2-3 days and we're clicktracking in IBM again!

(please feel free to edit the task description as I'm sure my verbiage isn't totally right...)

Event Timeline

CCogdill_WMF created this task.

@Jgreen or @cwdent do you have any thoughts here?

Nope, let's add the other teams to the task to start the security reviews as Caitlin said.

@Jgreen and @cwdent Can you please loop in the appropriate teams?

BBlack added subscribers: BBlack, Vgutierrez.

Yeah I do have concerns here. It's going to take some time before I can loop back and explain them, but I just wanted to put the note in now that this is concerning on multiple levels...

Updating task as I want to update the subdomain in the request.

Hi @BBlack - can you add your concerns to this ticket....we're needing to get this figured out soon. Thanks!

Bumping this! We are doing a series of newsletter tests with Chapters this quarter and it is really important for us to have access to click data to pages outside donate.wiki. We have already sent some emails out having to forego some data, but there are more going out over the next 4 weeks and it would be great to get what clicktracking we can. Not to mention there's a real human cost for the amount of time it takes to manually pull the data from another source.

Happy to answer any questions needed!

Dzahn changed the task status from Open to Stalled.Apr 17 2018, 5:48 PM

What's the data? From our clicktracking efforts what will we be collecting?

We're collecting click engagement off fundraising emails (actual
fundraising appeals, or informational newsletter emails) that are sent out
from the 3rd party email service provider we use. We want to use their
clicktracking system so we can get click data for sites outside of the
donate.wikimedia.org domain.

SSL certs are what allow your browser to show you a green bar and
guarantee that if you see that, you are talking to the Wikimedia
Foundation. If we allow a 3rd party to impersonate us using an SSL cert
it breaks that trust model. I personally think it's an important part
of making the internet safer for less technical people.

That being said, afaik there is no canonical rule that says it can't
mean "the Wikimedia Foundation and its trusted affiliates". If the
Certificate Authority (the org we buy them from) didn't want them to be
shared, it would say so in the terms of service. Of course it's
probably safe to assume their main motivation is profit.

Anyway I think the nit-picky definition is probably up to legal? I'm
sure this practice is entirely normal in a saas world. And CAs are an
absolute racket anyway so I am ambivalent.

Would conversion would be negatively affected by using a Silverpop URL
for the forwarding page? I would not be offended by that as a donor, I
feel like people know WMF is small and it's normal to farm out emails.

</$0.02>

@cwdent we formerly had silverpop-hosted urls in the email links, and lots of people thought they were phishing spam

We used a Silverpop URL for a few months and got enough complaints from
donors that our Donor Services team asked us to turn clicktracking off. We
didn't track clicks at all until fr-tech was able to build us a solution.

Thanks for the meeting on Thursday, everyone! I'm following up with IBM about potentially:

  • getting them to obtain a DV cert
  • reviewing their SSL setup

I'm still gathering information, but I think I got some helpful information regarding point #2. This is an SSL Report for another Trilogy client using a custom domain with IBM for the same purpose as we would be: https://www.ssllabs.com/ssltest/analyze.html?d=links.e.uso.org

@cwdent or @Jgreen would one of you check this out and let me know if it has enough info to determine if IBM can meet our HTTPS standards? I figure we should try to answer that question first.

The problems I see are:

  • content served over http
  • weak DH supported (https://weakdh.org/) resulting in "B" grade from Qualys

I don't think the 2nd is a blocker (yet, according to https://wikitech.wikimedia.org/wiki/HTTPS) but it sounds like the 1st is. @CCogdill_WMF can you see if they're willing to change their handling of http to redirect to https rather than serving a page?

Thanks Casey! I'm waiting for a reply. Just bumped it, FYI.

Sorry, I missed the above scan link earlier. The "weak DH" issue isn't mentioned explicitly on our policy page, but is definitely an issue. I think maybe the policy page wording is defective, but basically if ssllabs.com isn't showing an A+, it's a fail. I'll amend it a bit to be more explicit about that.

I reran the SSLLabs analyzer on links.e.uso.org today and it's still scored a B, looks like for several issue (still including weak DH).

@CCogdill_WMF @BBlack there's no change as far as I can see in Trilogy's SSL rating according to Qualys, still a B with the main issues being weak ciphers and weak key exchange. Is there any chance of them improving their security? How do we proceed if not?

I would choose not to proceed with a vendor who cares so little about security. The "Weak DH" issue, in particular, made security headlines back in 2015. Who knows what other security matters they've been ignoring for that long.

@Jgreen @BBlack thanks for bumping this and continuing to check. It does look like the SSL rating has bumped up to an A: https://www.ssllabs.com/ssltest/analyze.html?d=links.e.uso.org Though I'm not sure if that's using the same rating system as Qualys. Has anything changed from your end?

If not, I do hear your concerns. I would like to talk to relevant parties at fr-tech about setting up a way for us to perform easy redirects from donate wiki to other pages, such as the blog, so we can track clicks using our own systems.

@Jgreen @BBlack thanks for bumping this and continuing to check. It does look like the SSL rating has bumped up to an A: https://www.ssllabs.com/ssltest/analyze.html?d=links.e.uso.org Though I'm not sure if that's using the same rating system as Qualys. Has anything changed from your end?

If not, I do hear your concerns. I would like to talk to relevant parties at fr-tech about setting up a way for us to perform easy redirects from donate wiki to other pages, such as the blog, so we can track clicks using our own systems.

Qualys is the same company that hosts SSLLabs, and this is a major improvement. @BBlack do you see any remaining issues with this?

@BBlack circling back on this, do you still see any issue now after the Silverpop SSL improvements?

Checking back on this it looks like https://links.e.uso.org has slipped back to a B rating because they haven't ceased TLS 1.0/1.1 support. I'm not sure if that's still a valid hostname to check, or if this task is still relevant however.

Update! The link tracking domain removed support for TLS 1.0 and 1.1, so now score higher:

https://www.immuniweb.com/ssl/?id=de0FYntW

Can we revisit this?

Aklapper changed the task status from Stalled to Open.Nov 1 2020, 8:57 PM

Resetting task status per last comment.

Noting that the example domain (links.e.cantwell.com) also receives a score of A on Qualys SSL Labs test.

@JBennett @BBlack @Dwisehaupt @Jgreen I'm hearing that the email service provider (now branded acoustic) is getting higher ratings. What do you think of the situation now? Could we discuss this in the near future?

@JBennett @BBlack @Dwisehaupt @Jgreen I'm hearing that the email service provider (now branded acoustic) is getting higher ratings. What do you think of the situation now? Could we discuss this in the near future?

As of today Silverpop doesn't meet the published WMF HTTPS standard because they support weak ciphers plus some issues with the certificate chain (this latter part may not be relevant if we're buying/supplying the cert). I'm not aware of other cases where WMF provides an SSL cert to an external provider, so I don't know what an acceptable case looks like. But even if Silverpop fixes the cipher issues, we would probably need some assurance that they will consistently achieve a perfect score with SSL Labs since they have not done that at any point since this task was opened.

The swap of Traffic for Traffic-Icebox in this ticket's set of tags was based on a bulk action for all tickets that aren't are neither part of our current planned work nor clearly a recent, higher-priority emergent issue. This is simply one step in a larger task cleanup effort. Further triage of these tickets (and especially, organizing future potential project ideas from them into a new medium) will occur afterwards! For more detail, have a look at the extended explanation on the main page of Traffic-Icebox . Thank you!

Noting that Acoustic (née Silverpop) now uses SSL Certs hosted by Amazon Web Services and no longer supports weak ciphers. Here is an SSL Labs test of a current Acoustic client SSL cert: https://www.ssllabs.com/ssltest/analyze.html?d=links.e.protectus.org

@BBlack @KOfori Hi both! Could we bother you for another look at this task? A long time has passed since it originally was filed. The original reason for not pursuing this was that the ssl certs in use by Acoustic (née Silverpop) did not meet our standards (see eg T188561#4614934 and T188561#7402465).

The provider's ssllabs score is now an A (see https://www.ssllabs.com/ssltest/analyze.html?d=links.e.protectus.org, full report link https://www.ssllabs.com/ssltest/analyze.html?d=links.e.protectus.org&s=13.35.125.97)

Their TLS termination has been improved over time but they still don't meet the requirements listed on https://wikitech.wikimedia.org/wiki/HTTPS for a canonical domain (wikimedia.org):

  • They lack support of HSTS
  • They don't use CAA, at least for the example provided (liks.e.protectus.org)
  • They serve requests in plain text (again based on links.e.protectus.org)

Do we have access to some kind of documentation from the vendor (IBM's Watson Campaign Monitor, we call Silverpop) to be able to check if right now we can tune some of these settings?

Thanks for the feedback and requirements documentation, @Vgutierrez. Acoustic, the vendor in this case, doesn't have specific documentation regarding this, but I can tell you that they recently moved to certs issued by Amazon AWS. I am going to open a support ticket with Acoustic to see if they can address the specific issues you've noted. First, I wanted to ensure I understand them correctly so I have a couple of clarifying questions:

  • They lack support of HSTS

For this item, we need them to add an HSTS header to the pages that respond at the proposed domain (links.email.wikimedia.org), correct? And this header specification is sufficient: Strict-Transport-Security: max-age=31536000; includeSubDomains; preload?

  • They don't use CAA, at least for the example provided (liks.e.protectus.org)

I am not well-versed on DNS CAA implementation, but it looks like the key component is adding a CAA record to the domain's DNS that specifies the CAs that are allowed to issue certs for the domain. DNS in this case is controlled by Wikimedia. Is there anything Acoustic needs to implement to satisfy this requirement?

  • They serve requests in plain text (again based on links.e.protectus.org)

Clarifying that this is referring to the example link tracking domain (links.e.protectus.org) serving content over HTTP upon request rather than redirecting to HTTPS? If so, I can use the requirements provided to detail the implementation needs for Acoustic.

Thanks for the feedback and requirements documentation, @Vgutierrez. Acoustic, the vendor in this case, doesn't have specific documentation regarding this, but I can tell you that they recently moved to certs issued by Amazon AWS. I am going to open a support ticket with Acoustic to see if they can address the specific issues you've noted. First, I wanted to ensure I understand them correctly so I have a couple of clarifying questions:

  • They lack support of HSTS

For this item, we need them to add an HSTS header to the pages that respond at the proposed domain (links.email.wikimedia.org), correct? And this header specification is sufficient: Strict-Transport-Security: max-age=31536000; includeSubDomains; preload?

Yes, but the HSTS header must be set only on HTTPS requests, not for plain text ones

  • They don't use CAA, at least for the example provided (liks.e.protectus.org)

I am not well-versed on DNS CAA implementation, but it looks like the key component is adding a CAA record to the domain's DNS that specifies the CAs that are allowed to issue certs for the domain. DNS in this case is controlled by Wikimedia. Is there anything Acoustic needs to implement to satisfy this requirement?

Just let us know which CAs need to be authorized in our CAA record for links.email.wikimedia.org

  • They serve requests in plain text (again based on links.e.protectus.org)

Clarifying that this is referring to the example link tracking domain (links.e.protectus.org) serving content over HTTP upon request rather than redirecting to HTTPS? If so, I can use the requirements provided to detail the implementation needs for Acoustic.

Yes, that's it.