Page MenuHomePhabricator
Create Task
Maniphest T370821

Research and respond to Let's Encrypt's intent to deprecate OCSP in favour of CRLs
Closed, ResolvedPublic
Actions

Description

https://letsencrypt.org/2024/07/23/replacing-ocsp-with-crls.html

Today we are announcing our intent to end Online Certificate Status Protocol (OCSP) support in favor of Certificate Revocation Lists (CRLs) as soon as possible. OCSP and CRLs are both mechanisms by which CAs can communicate certificate revocation information, but CRLs have significant advantages over OCSP. Let’s Encrypt has been providing an OCSP responder since our launch nearly ten years ago. We added support for CRLs in 2022.

[...]

We recommend that anyone relying on OCSP services today start the process of ending that reliance as soon as possible.

We should look into this and start preparing for the transition in various places of our stack, in acme-chief and the CDN itself.

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
P:cache::haproxy and C:haproxy: remove OCSP flag and monitoringoperations/puppetproduction+1 -20
haproxy: Disable OCSP monitoring for LE unified certoperations/puppetproduction+10 -5
Remove now unused and obsolete LE OCSP health checkoperations/puppetproduction+0 -5
ncredir: Stop requiring OCSP on ssl monitoroperations/puppetproduction+1 -1
profile,cache: Stop monitoring OCSP freshness for acme-chief managed certsoperations/puppetproduction+1 -1
wikidough: Stop using OCSPoperations/puppetproduction+0 -1
apt: Remove OCSP staplingoperations/puppetproduction+0 -2
ncredir: Stop using OCSP staplingoperations/puppetproduction+0 -2
Customize query in gerrit

Related Objects

Event Timeline

ssingh created this task.Jul 23 2024, 8:05 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 23 2024, 8:05 PM
ssingh added a subscriber: Vgutierrez.Jul 23 2024, 8:06 PM
BBlack subscribed.Jul 23 2024, 8:12 PM

Firefox has historically been the reason we've been stapling OCSP for the past many years. If our certificate has an OCSP URI in its metadata, then Firefox will check OCSP in realtime (which is a privacy risk) unless our servers staple the OCSP to the TLS negotiation (which we do!). This applies to both our Digicert and LE unified certs (and I'm sure some other lesser cases as well!).

Basically the transition goes something like this for each CA:

  • The CA stops adding OCSP URI to new certs they issue us.
  • We deploy such a cert, which both makes stapling unnecessary for that cert, but which breaks our ocsp fetching/caching/stapling stuff for that cert.
  • We disable the fetching/caching/stapling for that cert
BBlack added a comment.Jul 23 2024, 8:14 PM

Note also Digicert's annual renewal is coming soon in T368560 . We should maybe look at whether the OCSP URI is optional in the form for making the cert, and turn it off (assuming they also have CRLs working fine). Or if they're not ready for this, I guess Digicert waits another year.

Vgutierrez changed the task status from Open to In Progress.May 15 2025, 10:56 AM
Vgutierrez triaged this task as Unbreak Now! priority.
Vgutierrez moved this task from Backlog to Actively Servicing on the Traffic board.

Let's encrypt already stopped including OCSP urls in new certificates and it's already causing issues in production:

root@acmechief2002:~# openssl x509 -dates -issuer -ocsp_uri -noout -in /var/lib/acme-chief/certs/non-canonical-redirect-3/live/ec-prime256v1.crt
notBefore=May  8 01:49:48 2025 GMT
notAfter=Aug  6 01:49:47 2025 GMT
issuer=C = US, O = Let's Encrypt, CN = E6
root@acmechief2002:~# openssl x509 -dates -ocsp_uri -issuer -noout -in /var/lib/acme-chief/certs/non-canonical-redirect-5/live/ec-prime256v1.crt
notBefore=May  6 05:51:39 2025 GMT
notAfter=Aug  4 05:51:38 2025 GMT
http://e5.o.lencr.org
issuer=C = US, O = Let's Encrypt, CN = E5

from https://letsencrypt.org/2024/12/05/ending-ocsp/:

January 30, 2025
OCSP Must-Staple requests will fail, unless the requesting account has previously issued a certificate containing the OCSP Must Staple extension
May 7, 2025
Prior to this date we will have added CRL URLs to certificates
On this date we will drop OCSP URLs from certificates
On this date all requests including the OCSP Must Staple extension will fail
August 6, 2025
On this date we will turn off our OCSP responders

gerritbot added a comment.May 15 2025, 11:02 AM

Change #1146550 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] profile,cache: Stop monitoring OCSP freshness for acme-chief managed certs

https://gerrit.wikimedia.org/r/1146550

gerritbot added a project: Patch-For-Review.May 15 2025, 11:02 AM
gerritbot added a comment.May 15 2025, 11:06 AM

Change #1146552 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] ncredir: Stop using OCSP stapling

https://gerrit.wikimedia.org/r/1146552

gerritbot added a comment.May 15 2025, 11:11 AM

Change #1146555 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] apt: Remove OCSP stapling

https://gerrit.wikimedia.org/r/1146555

gerritbot added a comment.May 15 2025, 11:11 AM

Change #1146556 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] wikidough: Stop using OCSP

https://gerrit.wikimedia.org/r/1146556

gerritbot added a comment.May 15 2025, 11:12 AM

Change #1146552 merged by Vgutierrez:

[operations/puppet@production] ncredir: Stop using OCSP stapling

https://gerrit.wikimedia.org/r/1146552

gerritbot added a comment.May 15 2025, 11:16 AM

Change #1146555 merged by Muehlenhoff:

[operations/puppet@production] apt: Remove OCSP stapling

https://gerrit.wikimedia.org/r/1146555

gerritbot added a comment.May 15 2025, 11:17 AM

Change #1146556 merged by Vgutierrez:

[operations/puppet@production] wikidough: Stop using OCSP

https://gerrit.wikimedia.org/r/1146556

Stashbot added a comment.May 15 2025, 11:21 AM

Mentioned in SAL (#wikimedia-operations) [2025-05-15T11:21:02Z] <sukhe> sudo cumin -b1 -s10 "A:wikidough" "run-puppet-agent": T370821

gerritbot added a comment.May 15 2025, 11:25 AM

Change #1146559 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] ncredir: Stop requiring OCSP on ssl monitor

https://gerrit.wikimedia.org/r/1146559

gerritbot added a comment.May 15 2025, 11:27 AM

Change #1146550 merged by Vgutierrez:

[operations/puppet@production] profile,cache: Stop monitoring OCSP freshness for acme-chief managed certs

https://gerrit.wikimedia.org/r/1146550

gerritbot added a comment.May 15 2025, 11:30 AM

Change #1146559 merged by Vgutierrez:

[operations/puppet@production] ncredir: Stop requiring OCSP on ssl monitor

https://gerrit.wikimedia.org/r/1146559

Vgutierrez lowered the priority of this task from Unbreak Now! to High.May 15 2025, 11:32 AM
gerritbot added a comment.May 15 2025, 11:41 AM

Change #1146563 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Remove now unused and obsolete LE OCSP health check

https://gerrit.wikimedia.org/r/1146563

gerritbot added a comment.May 15 2025, 12:39 PM

Change #1146563 merged by Muehlenhoff:

[operations/puppet@production] Remove now unused and obsolete LE OCSP health check

https://gerrit.wikimedia.org/r/1146563

Maintenance_bot removed a project: Patch-For-Review.May 15 2025, 1:30 PM
gerritbot added a comment.Jun 19 2025, 8:10 AM

Change #1161397 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] haproxy: Disable OCSP monitoring for LE unified cert

https://gerrit.wikimedia.org/r/1161397

gerritbot added a project: Patch-For-Review.Jun 19 2025, 8:10 AM
gerritbot added a comment.Jun 19 2025, 8:46 AM

Change #1161397 merged by Vgutierrez:

[operations/puppet@production] haproxy: Disable OCSP monitoring for LE unified cert

https://gerrit.wikimedia.org/r/1161397

Vgutierrez closed this task as Resolved.Jun 19 2025, 8:51 AM
Vgutierrez claimed this task.
Maintenance_bot removed a project: Patch-For-Review.Jun 19 2025, 9:31 AM
gerritbot added a comment.Jul 9 2025, 4:32 PM

Change #1167674 had a related patch set uploaded (by Ssingh; author: Ssingh):

[operations/puppet@production] P:cache::haproxy and C:haproxy: remove OCSP flag and monitoring

https://gerrit.wikimedia.org/r/1167674

gerritbot added a project: Patch-For-Review.Jul 9 2025, 4:32 PM
gerritbot added a comment.Jul 9 2025, 5:35 PM

Change #1167674 abandoned by Ssingh:

[operations/puppet@production] P:cache::haproxy and C:haproxy: remove OCSP flag and monitoring

Reason:

splitting this up.

https://gerrit.wikimedia.org/r/1167674

ssingh mentioned this in T399114: Remove OCSP monitoring and related bits.Jul 9 2025, 6:22 PM
Maintenance_bot removed a project: Patch-For-Review.Jul 9 2025, 6:30 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits