Page MenuHomePhabricator

SSL cert for links.email.wikimedia.org
Closed, ResolvedPublic

Description

Back in 2015, we turned off our email service provider's (IBM's Watson Campaign Monitor, we call Silverpop) clicktracking feature because the only way we were able to track clicks on a custom domain was over http. The Foundation was moving to adopt an HSTS policy and couldn't have unsecured domains out there. So we started using landing page hits on donate.wiki as an approximation for clicks. That has worked well enough, but has three limitations/pain points:

a) It means we have to get our email performance data from 3 different sources, rather than 2, which slows down the stats process.

b) We aren't able to track clicks to any link not hosted on donate.wikimedia.org

c) We don't have individual clickthrough events recorded in the ESP so it's harder to set up behavioral email programs

Recently, Silverpop changed their policy and are allowing clients to purchase SSL certs for clicktracking domains, so we'd like to switch back to using our ESP for clicktracking. We need to get Ops' approval to use a subdomain for clicktracking again so we can then have them purchase a cert. For now, we need to confirm this information is correct:

2-Character Country: US
State: California
Locality/City: San Francisco
Organization Name (Company Name): Wikimedia Foundation, Inc.
Organization Unit (example: Marketing, Sales, etc): Fundraising
Common Name: links.email.donate.wikimedia.org (this is the subdomain we used before)
Contact Email Address: [REDACTED]

Silverpop will then provide a CSR for us to purchase the cert with. It's recommended that we buy a 3-year cert. As to what level, that is up to Ops, but the user virtually never sees this site load so we don't think we need to spend much on it. Then we provide will then provide the cert to Trilogy and they will provide it to IBM. After that it takes just 2-3 days and we're clicktracking in IBM again!

(please feel free to edit the task description as I'm sure my verbiage isn't totally right...)

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

The problems I see are:

  • content served over http
  • weak DH supported (https://weakdh.org/) resulting in "B" grade from Qualys

I don't think the 2nd is a blocker (yet, according to https://wikitech.wikimedia.org/wiki/HTTPS) but it sounds like the 1st is. @CCogdill_WMF can you see if they're willing to change their handling of http to redirect to https rather than serving a page?

Thanks Casey! I'm waiting for a reply. Just bumped it, FYI.

Sorry, I missed the above scan link earlier. The "weak DH" issue isn't mentioned explicitly on our policy page, but is definitely an issue. I think maybe the policy page wording is defective, but basically if ssllabs.com isn't showing an A+, it's a fail. I'll amend it a bit to be more explicit about that.

I reran the SSLLabs analyzer on links.e.uso.org today and it's still scored a B, looks like for several issue (still including weak DH).

@CCogdill_WMF @BBlack there's no change as far as I can see in Trilogy's SSL rating according to Qualys, still a B with the main issues being weak ciphers and weak key exchange. Is there any chance of them improving their security? How do we proceed if not?

I would choose not to proceed with a vendor who cares so little about security. The "Weak DH" issue, in particular, made security headlines back in 2015. Who knows what other security matters they've been ignoring for that long.

@Jgreen @BBlack thanks for bumping this and continuing to check. It does look like the SSL rating has bumped up to an A: https://www.ssllabs.com/ssltest/analyze.html?d=links.e.uso.org Though I'm not sure if that's using the same rating system as Qualys. Has anything changed from your end?

If not, I do hear your concerns. I would like to talk to relevant parties at fr-tech about setting up a way for us to perform easy redirects from donate wiki to other pages, such as the blog, so we can track clicks using our own systems.

@Jgreen @BBlack thanks for bumping this and continuing to check. It does look like the SSL rating has bumped up to an A: https://www.ssllabs.com/ssltest/analyze.html?d=links.e.uso.org Though I'm not sure if that's using the same rating system as Qualys. Has anything changed from your end?

If not, I do hear your concerns. I would like to talk to relevant parties at fr-tech about setting up a way for us to perform easy redirects from donate wiki to other pages, such as the blog, so we can track clicks using our own systems.

Qualys is the same company that hosts SSLLabs, and this is a major improvement. @BBlack do you see any remaining issues with this?

@BBlack circling back on this, do you still see any issue now after the Silverpop SSL improvements?

Checking back on this it looks like https://links.e.uso.org has slipped back to a B rating because they haven't ceased TLS 1.0/1.1 support. I'm not sure if that's still a valid hostname to check, or if this task is still relevant however.

Update! The link tracking domain removed support for TLS 1.0 and 1.1, so now score higher:

https://www.immuniweb.com/ssl/?id=de0FYntW

Can we revisit this?

Aklapper changed the task status from Stalled to Open.Nov 1 2020, 8:57 PM

Resetting task status per last comment.

Noting that the example domain (links.e.cantwell.com) also receives a score of A on Qualys SSL Labs test.

@JBennett @BBlack @Dwisehaupt @Jgreen I'm hearing that the email service provider (now branded acoustic) is getting higher ratings. What do you think of the situation now? Could we discuss this in the near future?

@JBennett @BBlack @Dwisehaupt @Jgreen I'm hearing that the email service provider (now branded acoustic) is getting higher ratings. What do you think of the situation now? Could we discuss this in the near future?

As of today Silverpop doesn't meet the published WMF HTTPS standard because they support weak ciphers plus some issues with the certificate chain (this latter part may not be relevant if we're buying/supplying the cert). I'm not aware of other cases where WMF provides an SSL cert to an external provider, so I don't know what an acceptable case looks like. But even if Silverpop fixes the cipher issues, we would probably need some assurance that they will consistently achieve a perfect score with SSL Labs since they have not done that at any point since this task was opened.

The swap of Traffic for Traffic-Icebox in this ticket's set of tags was based on a bulk action for all tickets that aren't are neither part of our current planned work nor clearly a recent, higher-priority emergent issue. This is simply one step in a larger task cleanup effort. Further triage of these tickets (and especially, organizing future potential project ideas from them into a new medium) will occur afterwards! For more detail, have a look at the extended explanation on the main page of Traffic-Icebox . Thank you!

Noting that Acoustic (née Silverpop) now uses SSL Certs hosted by Amazon Web Services and no longer supports weak ciphers. Here is an SSL Labs test of a current Acoustic client SSL cert: https://www.ssllabs.com/ssltest/analyze.html?d=links.e.protectus.org

@BBlack @KOfori Hi both! Could we bother you for another look at this task? A long time has passed since it originally was filed. The original reason for not pursuing this was that the ssl certs in use by Acoustic (née Silverpop) did not meet our standards (see eg T188561#4614934 and T188561#7402465).

The provider's ssllabs score is now an A (see https://www.ssllabs.com/ssltest/analyze.html?d=links.e.protectus.org, full report link https://www.ssllabs.com/ssltest/analyze.html?d=links.e.protectus.org&s=13.35.125.97)

Their TLS termination has been improved over time but they still don't meet the requirements listed on https://wikitech.wikimedia.org/wiki/HTTPS for a canonical domain (wikimedia.org):

  • They lack support of HSTS
  • They don't use CAA, at least for the example provided (liks.e.protectus.org)
  • They serve requests in plain text (again based on links.e.protectus.org)

Do we have access to some kind of documentation from the vendor (IBM's Watson Campaign Monitor, we call Silverpop) to be able to check if right now we can tune some of these settings?

Thanks for the feedback and requirements documentation, @Vgutierrez. Acoustic, the vendor in this case, doesn't have specific documentation regarding this, but I can tell you that they recently moved to certs issued by Amazon AWS. I am going to open a support ticket with Acoustic to see if they can address the specific issues you've noted. First, I wanted to ensure I understand them correctly so I have a couple of clarifying questions:

  • They lack support of HSTS

For this item, we need them to add an HSTS header to the pages that respond at the proposed domain (links.email.wikimedia.org), correct? And this header specification is sufficient: Strict-Transport-Security: max-age=31536000; includeSubDomains; preload?

  • They don't use CAA, at least for the example provided (liks.e.protectus.org)

I am not well-versed on DNS CAA implementation, but it looks like the key component is adding a CAA record to the domain's DNS that specifies the CAs that are allowed to issue certs for the domain. DNS in this case is controlled by Wikimedia. Is there anything Acoustic needs to implement to satisfy this requirement?

  • They serve requests in plain text (again based on links.e.protectus.org)

Clarifying that this is referring to the example link tracking domain (links.e.protectus.org) serving content over HTTP upon request rather than redirecting to HTTPS? If so, I can use the requirements provided to detail the implementation needs for Acoustic.

Thanks for the feedback and requirements documentation, @Vgutierrez. Acoustic, the vendor in this case, doesn't have specific documentation regarding this, but I can tell you that they recently moved to certs issued by Amazon AWS. I am going to open a support ticket with Acoustic to see if they can address the specific issues you've noted. First, I wanted to ensure I understand them correctly so I have a couple of clarifying questions:

  • They lack support of HSTS

For this item, we need them to add an HSTS header to the pages that respond at the proposed domain (links.email.wikimedia.org), correct? And this header specification is sufficient: Strict-Transport-Security: max-age=31536000; includeSubDomains; preload?

Yes, but the HSTS header must be set only on HTTPS requests, not for plain text ones

  • They don't use CAA, at least for the example provided (liks.e.protectus.org)

I am not well-versed on DNS CAA implementation, but it looks like the key component is adding a CAA record to the domain's DNS that specifies the CAs that are allowed to issue certs for the domain. DNS in this case is controlled by Wikimedia. Is there anything Acoustic needs to implement to satisfy this requirement?

Just let us know which CAs need to be authorized in our CAA record for links.email.wikimedia.org

  • They serve requests in plain text (again based on links.e.protectus.org)

Clarifying that this is referring to the example link tracking domain (links.e.protectus.org) serving content over HTTP upon request rather than redirecting to HTTPS? If so, I can use the requirements provided to detail the implementation needs for Acoustic.

Yes, that's it.

Hey ya'll, just making sure there's nothing for us on FR-Tech to do right now. I assume it's still in "Let the Email team and Traffic team work out some details", but let me know if that's wrong!

Correct, @greg. I'm still gathering feedback from Acoustic's support and will report back here. Hoping to get more information this week.

An update: Acoustic has confirmed they can implement all of the requirements mentioned in this comment on this task. The example click tracking domain - links.e.protectus.org - has been updated to meet these requirements in order to show this in practice and it receives an A+ rating on SSL Labs. For implementing DNS CAA, the CA is Amazon (amazon.com).

With this update, are we able to move forward with this task of issuing an SSL cert for links.email.wikimedia.org in order to use this subdomain for link tracking? The original description in this task of next steps is likely out of date and I am confirming with Acoustic what is needed on their end to complete this process.

I can confirm that they've added HSTS support and stopped serving traffic in port 80 and redirect it to port 443:

$ curl -I links.e.protectus.org -s |grep -i location
Location: https://links.e.protectus.org/
$ curl -I -s https://links.e.protectus.org |grep strict-transport-security
strict-transport-security: max-age=31536000; includeSubDomains; preload

Hey. Everyone. Where is this exactly. It would be good to get this click-through tracking done very soon. What would it take to get this rolling in early Q3. Maybe for the Thank You campaign in early Jan.

Hi all,

Noting that I added a comment with an action request to make DNS updates on wikimedia.org to the parent task. I added it to that task as the request is not specifically related to our SSL security discussion in this subtask.

@Vgutierrez is there anything left to do so that we can move forward on this task? Please let me know as we are trying to get this up and running as quickly as possible.

@KOfori can you inform me as to where this falls in the list of your teams priorities? Any ETA on completion? I know this task, in one shape or another, has been open for quite some time. I wanted to see if we could get click-tracking turned on in our email service provider prior to an email dropping on 12/27/22.

@Vgutierrez is there anything left to do so that we can move forward on this task? Please let me know as we are trying to get this up and running as quickly as possible.

@DBu-WMF All good from our side. Just let us know the A/AAAA records needed on links.email.wikimedia.org

@greg can we move forward and turn click-tracking back on in Acoustic?

Regarding DNS updates, I am going to paste the comment I linked to in my last comment below so all of the information is in this phab task:

To move forward with click tracking, Acoustic needs us to implement several DNS updates (attached). In order to use a custom click tracking subdomain, Acoustic requires a full custom domain setup. The other subdomains are for for reply, open, and bounce handling. These are currently used but at generic domains (ex. bounce.wikimedia.mkt4477.com for bounce handling), these DNS entries will mask them behind a custom domain. This is Trilogy’s recommended setup for all clients.

Here are more details from Acoustic regarding the setup and benefit of the custom domain: https://help.goacoustic.com/hc/en-us/articles/360042763214-Provision-a-custom-domain-for-email-use

Change 868103 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/dns@master] wikimedia.org: Add links.email related DNS records

https://gerrit.wikimedia.org/r/868103

I've created a CR https://gerrit.wikimedia.org/r/c/operations/dns/+/868103 to add the DNS records to the wikimedia.org DNS zone. Besides the DNS records listed on the file attached by @EWilfong_WMF I've set a CAA record on email.wikimedia.org to allow Amazon and Let's Encrypt to issue certs

Change 868103 merged by Vgutierrez:

[operations/dns@master] wikimedia.org: Add links.email related DNS records

https://gerrit.wikimedia.org/r/868103

DNS records are now live:

$ host -t cname links.email.wikimedia.org
links.email.wikimedia.org is an alias for recp.mkt41.net.

let me know if you need anything else from our side @EWilfong_WMF

Thank you, @Vgutierrez. I've alerted Acoustic support that the updates have been made and I will follow up here when they provide next steps.

@Vgutierrez - Attached are the DNS updates needed to verify the SSL certs. Some will replace the existing CNAMEs. Once these are live, we should be done with the DNS updates for this task.

@Vgutierrez sorry to be a pain but if there is any possibility that we can get this done quickly it would be amazing. We have the largest email we have ever sent dropping on 12/27 and if we can click-tracking live for that email I will die happy.

Change 869277 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/dns@master] wikimedia.org: Add cert validation records for links.email

https://gerrit.wikimedia.org/r/869277

Change 869277 merged by Ssingh:

[operations/dns@master] wikimedia.org: Add cert validation records for links.email

https://gerrit.wikimedia.org/r/869277

Mentioned in SAL (#wikimedia-operations) [2022-12-19T18:29:05Z] <sukhe> running authdns-update for Gerrit: 869277: T188561

@Vgutierrez sorry to be a pain but if there is any possibility that we can get this done quickly it would be amazing. We have the largest email we have ever sent dropping on 12/27 and if we can click-tracking live for that email I will die happy.

We have merged the changes and they are now live. Let us know if you need something else from our end.

Thank you, @ssingh. These changes look good to me and I am asking Acoustic to verify. @DBu-WMF, Brian Sisolak and I will follow up with next steps and testing of the link tracking once Acoustic completes their setup.

Noting that the cert for links.email is now in place and all requirements stated in this task are implemented: https://www.ssllabs.com/ssltest/analyze.html?d=links.email.wikimedia.org

From my perspective, this task can be closed unless there are any further questions or comments about the SSL cert for this subdomain.

(Assigning to @Vgutierrez per the work/patch, letting ya'll review/close however you'd like over in Traffic.)