Page MenuHomePhabricator

Remove RSA certificates and use only ECDSA certificates
Open, In Progress, MediumPublic

Description

RSA certificate usage comprises only a tiny fraction of traffic vs ECDSA:

SELECT tls_auth, COUNT(tls_auth) from "druid"."webrequest_sampled_live"
  WHERE __time >= TIMESTAMP '2024-06-23'
  GROUP by tls_auth;

yields:

ECDSA2416766928
RSA2388370

In the last month, RSA traffic accounted for ~0.1% of traffic.

Very low demand, ever-weakening security, and increased cost/overhead of certificate issuance (WMF pays double so that we get RSA certificates) tells us that this is a good time to remove RSA certificates and only use ECDSA certificates.

Event Timeline

BCornwall triaged this task as Medium priority.Jul 24 2024, 12:26 AM
BCornwall moved this task from Backlog to Scheduled incidental work on the Traffic board.
BCornwall renamed this task from Remove RSA certificate support to Remove RSA certificates and use only ECDSA certificates.Sep 3 2024, 10:10 PM
BCornwall updated the task description. (Show Details)

Change #1072590 had a related patch set uploaded (by BCornwall; author: BCornwall):

[operations/puppet@production] Remove RSA certificate support

https://gerrit.wikimedia.org/r/1072590

I just -2ed the gerrit change cause we don't currently have information about which certificate is being used.
TLSv1.2 includes the authentication mechanism used during the handshake as part of the ciphersuite (ECDHE-RSA-AES256-GCM-SHA384, ECDHE-ECDSA-AES256-GCM-SHA384) and that's what's currently being sent from haproxy to varnish as part of x-connection-properties header.
In TLSv1.3 he authentication mechanism has been dropped from the ciphersuite name (TLS_AES_128_GCM_SHA256 could be used with either ECDSA or RSA certificates).

To make things even worse, our current VCL code assumes that if RSA can't be found as part of the ciphersuite name, ECDSA must be being used:

set req.http.x-tls-auth = regsub(req.http.X-Connection-Properties, ".* C=([^; ]+);.*", "\1");
set req.http.x-tls-ciph = regsub(req.http.x-tls-auth, "^(ECDHE-(ECDSA|RSA)|DHE-RSA)-", ""); 
if (req.http.x-tls-auth ~ "^ECDHE-RSA") {
    set req.http.x-tls-auth = "RSA";
} else if (req.http.x-tls-auth ~ "^DHE-RSA") {
    set req.http.x-tls-auth = "RSA";
    set req.http.x-tls-keyx = "DHE";
} else {
    set req.http.x-tls-auth = "ECDSA";
}

If you consider that during the last 24 hours from the 85 million of TLS requests tracked by turnilo, only a 3.47% are using TLSv1.2, so that means that we only have RSA usage data from a 3.47% of the requests that we are currently getting

Now that some time has passed since the misreporting has been fixed I checked again:

SELECT tls_auth, COUNT(tls_auth) from "druid"."webrequest_sampled_live"
  WHERE __time >= TIMESTAMP '2024-09-28'
  GROUP by tls_auth;
ECDSA488182110
RSA959153

~0.2% of traffic is using RSA by these metrics.

Change #1078468 had a related patch set uploaded (by BCornwall; author: BCornwall):

[operations/puppet@production] haproxy: Reorder acmecerts to demote rsa-2048

https://gerrit.wikimedia.org/r/1078468

Change #1078468 merged by BCornwall:

[operations/puppet@production] haproxy: Reorder acmecerts to demote rsa-2048

https://gerrit.wikimedia.org/r/1078468

Change #1072590 merged by BCornwall:

[operations/puppet@production] varnish: Give 1% of views RSA cert warnings

https://gerrit.wikimedia.org/r/1072590

BCornwall changed the task status from Open to In Progress.Oct 23 2024, 8:42 PM

Change #1084855 had a related patch set uploaded (by BCornwall; author: BCornwall):

[operations/puppet@production] varnish: Increase RSA cert warnings to 2% of views

https://gerrit.wikimedia.org/r/1084855

Change #1084855 merged by BCornwall:

[operations/puppet@production] varnish: Increase RSA cert warnings to 2% of views

https://gerrit.wikimedia.org/r/1084855

Change #1087954 had a related patch set uploaded (by BCornwall; author: BCornwall):

[operations/puppet@production] varnish: Increase RSA cert warnings to 5% of views

https://gerrit.wikimedia.org/r/1087954

Change #1087954 merged by BCornwall:

[operations/puppet@production] varnish: Increase RSA cert warnings to 5% of views

https://gerrit.wikimedia.org/r/1087954

Change #1090912 had a related patch set uploaded (by BCornwall; author: BCornwall):

[operations/puppet@production] varnish: Increase RSA cert warnings to 10%

https://gerrit.wikimedia.org/r/1090912

Change #1090912 merged by BCornwall:

[operations/puppet@production] varnish: Increase RSA cert warnings to 10%

https://gerrit.wikimedia.org/r/1090912

Change #1097446 had a related patch set uploaded (by BCornwall; author: BCornwall):

[operations/puppet@production] varnish: Increase RSA cert warnings to 100%

https://gerrit.wikimedia.org/r/1097446

Change #1097446 merged by BCornwall:

[operations/puppet@production] varnish: Increase RSA cert warnings to 100%

https://gerrit.wikimedia.org/r/1097446

Quiddity subscribed.

Hi, I believe this change probably deserves an entry in Tech News. The last similar change that I'm aware of, was announced using this wording (below).
Please could someone help me by writing a similar entry, for this upcoming change?

You can no longer read Wikimedia wikis if your browser use old TLS versions. This is because it is a security problem for everyone. It can lead to downgrade attacks. Since October 29, 2020, users who use old TLS versions will not be able to connect to Wikimedia projects. A list of recommended browsers is available. All modern operating systems and browsers are always able to reach Wikimedia projects."

Thanks.

@Quiddity There's some verbiage on https://en.wikipedia.org/sec-warning that you could use, e.g.:

Wikimedia projects, including Wikipedia, are getting more secure.

We are removing support for RSA-based HTTPS certificates - specifically rsa-2048. Some older browser software relies on this to connect to our sites. Certain older, unsupported browser or smartphone devices without this support will be unable to connect; Instead, they will display a connectivity error.

See the HTTPS Browser Recommendations page for more-detailed information. Please update your device or contact your IT administrator for any concerns.

Thank you! For the record (or in case edits are needed before it is frozen on Friday), I've added it to https://meta.wikimedia.org/wiki/Tech/News/2024/49 using the wording:

Starting this week, Wikimedia wikis no longer support connections using old RSA-based HTTPS certificates, specifically rsa-2048. This change is to improve security for all users. Some older, unsupported browser or smartphone devices will be unable to connect; Instead, they will display a connectivity error. See the HTTPS Browser Recommendations page for more-detailed information. All modern operating systems and browsers are always able to reach Wikimedia projects. (1)