Page MenuHomePhabricator

Let's Encrypt transitioning to ISRG's Root
Open, MediumPublic

Description

According to https://community.letsencrypt.org/t/transition-to-isrgs-root-delayed-until-sep-29/125516 on September 29th Let's Encrypt will transition to ISRG's Root CA, this will impede users of Android <7.1 to access Wikipedia on eqiad, codfw and ulsfo (the current DCs using the LE version of the unified cert).

Let's Encrypt is currently providing an alternative relation chain to mitigate the issue, to benefit from this feature we need to backport https://github.com/certbot/certbot/pull/8080/commits/5c47ab8d2a19c5d07ff8f538314e7d5f34fdd496 on top of python3-acme 0.31.0-2 and then use the alternate link to fetch the issued certificate on acme-chief

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald Transcript
ema triaged this task as High priority.Sep 16 2020, 8:48 AM
ema moved this task from Triage to TLS on the Traffic board.

I have backported the patch to python-acme-0.31.0, here it is for review and future reference.

Description: Backport of upstream patch "certbot: add --preferred-chain"
Author: Emanuele Rocca <ema@wikimedia.org>
Bug-WMF: https://phabricator.wikimedia.org/T263006

---
Origin: upstream, https://github.com/certbot/certbot/pull/8080/commits/5c47ab8d2a19c5d07ff8f538314e7d5f34fdd496
Bug: https://github.com/certbot/certbot/issues/6971
Last-Update: 2020-09-16

--- python-acme-0.31.0.orig/acme/client.py
+++ python-acme-0.31.0/acme/client.py
@@ -15,6 +15,7 @@ import re
 from requests_toolbelt.adapters.source import SourceAddressAdapter
 import requests
 from requests.adapters import HTTPAdapter
+from requests.utils import parse_header_links
 import sys
 
 from acme import challenges
@@ -738,11 +739,13 @@ class ClientV2(ClientBase):
             raise errors.ValidationError(failed)
         return orderr.update(authorizations=responses)
 
-    def finalize_order(self, orderr, deadline):
+    def finalize_order(self, orderr, deadline, fetch_alternative_chains=False):
         """Finalize an order and obtain a certificate.
 
         :param messages.OrderResource orderr: order to finalize
         :param datetime.datetime deadline: when to stop polling and timeout
+        :param bool fetch_alternative_chains: whether to also fetch alternative
+            certificate chains
 
         :returns: finalized order
         :rtype: messages.OrderResource
@@ -759,8 +762,13 @@ class ClientV2(ClientBase):
             if body.error is not None:
                 raise errors.IssuanceError(body.error)
             if body.certificate is not None:
-                certificate_response = self._post_as_get(body.certificate).text
-                return orderr.update(body=body, fullchain_pem=certificate_response)
+                certificate_response = self._post_as_get(body.certificate)
+                orderr = orderr.update(body=body, fullchain_pem=certificate_response.text)
+                if fetch_alternative_chains:
+                    alt_chains_urls = self._get_links(certificate_response, 'alternate')
+                    alt_chains = [self._post_as_get(url).text for url in alt_chains_urls]
+                    orderr = orderr.update(alternative_fullchains_pem=alt_chains)
+                return orderr
         raise errors.TimeoutError()
 
     def revoke(self, cert, rsn):
@@ -809,6 +817,20 @@ class ClientV2(ClientBase):
         # If POST-as-GET is not supported yet, we use a GET instead.
         return self.net.get(*args, **kwargs)
 
+    def _get_links(self, response, relation_type):
+        """
+        Retrieves all Link URIs of relation_type from the response.
+        :param requests.Response response: The requests HTTP response.
+        :param str relation_type: The relation type to filter by.
+        """
+        # Can't use response.links directly because it drops multiple links
+        # of the same relation type, which is possible in RFC8555 responses.
+        if not 'Link' in response.headers:
+            return []
+        links = parse_header_links(response.headers['Link'])
+        return [l['url'] for l in links
+                if 'rel' in l and 'url' in l and l['rel'] == relation_type]
+
 
 class BackwardsCompatibleClientV2(object):
     """ACME client wrapper that tends towards V2-style calls, but
@@ -888,11 +910,13 @@ class BackwardsCompatibleClientV2(object
         else:
             return self.client.new_order(csr_pem)
 
-    def finalize_order(self, orderr, deadline):
+    def finalize_order(self, orderr, deadline, fetch_alternative_chains=False):
         """Finalize an order and obtain a certificate.
 
         :param messages.OrderResource orderr: order to finalize
         :param datetime.datetime deadline: when to stop polling and timeout
+        :param bool fetch_alternative_chains: whether to also fetch alternative
+            certificate chains
 
         :returns: finalized order
         :rtype: messages.OrderResource
@@ -924,7 +948,7 @@ class BackwardsCompatibleClientV2(object
 
             return orderr.update(fullchain_pem=(cert + chain))
         else:
-            return self.client.finalize_order(orderr, deadline)
+            return self.client.finalize_order(orderr, deadline, fetch_alternative_chains)
 
     def revoke(self, cert, rsn):
         """Revoke certificate.
--- python-acme-0.31.0.orig/acme/client_test.py
+++ python-acme-0.31.0/acme/client_test.py
@@ -262,7 +262,7 @@ class BackwardsCompatibleClientV2Test(Cl
         with mock.patch('acme.client.ClientV2') as mock_client:
             client = self._init()
             client.finalize_order(mock_orderr, mock_deadline)
-            mock_client().finalize_order.assert_called_once_with(mock_orderr, mock_deadline)
+            mock_client().finalize_order.assert_called_once_with(mock_orderr, mock_deadline, False)
 
     def test_revoke(self):
         self.response.json.return_value = DIRECTORY_V1.to_json()
@@ -864,6 +864,32 @@ class ClientV2Test(ClientTestBase):
         deadline = datetime.datetime.now() - datetime.timedelta(seconds=60)
         self.assertRaises(errors.TimeoutError, self.client.finalize_order, self.orderr, deadline)
 
+    def test_finalize_order_alt_chains(self):
+        updated_order = self.order.update(
+            certificate='https://www.letsencrypt-demo.org/acme/cert/',
+        )
+        updated_orderr = self.orderr.update(body=updated_order,
+                                            fullchain_pem=CERT_SAN_PEM,
+                                            alternative_fullchains_pem=[CERT_SAN_PEM,
+                                                                        CERT_SAN_PEM])
+        self.response.json.return_value = updated_order.to_json()
+        self.response.text = CERT_SAN_PEM
+        self.response.headers['Link'] ='<https://example.com/acme/cert/1>;rel="alternate", ' + \
+            '<https://exaple.com/dir>;rel="index", ' + \
+            '<https://example.com/acme/cert/2>;title="foo";rel="alternate"'
+
+        deadline = datetime.datetime(9999, 9, 9)
+        resp = self.client.finalize_order(self.orderr, deadline, fetch_alternative_chains=True)
+        self.net.post.assert_any_call('https://example.com/acme/cert/1',
+                                      mock.ANY, acme_version=2, new_nonce_url=mock.ANY)
+        self.net.post.assert_any_call('https://example.com/acme/cert/2',
+                                      mock.ANY, acme_version=2, new_nonce_url=mock.ANY)
+        self.assertEqual(resp, updated_orderr)
+
+        del self.response.headers['Link']
+        resp = self.client.finalize_order(self.orderr, deadline, fetch_alternative_chains=True)
+        self.assertEqual(resp, updated_orderr.update(alternative_fullchains_pem=[]))
+
     def test_revoke(self):
         self.client.revoke(messages_test.CERT, self.rsn)
         self.net.post.assert_called_once_with(
--- python-acme-0.31.0.orig/acme/messages.py
+++ python-acme-0.31.0/acme/messages.py
@@ -573,11 +573,15 @@ class OrderResource(ResourceWithURI):
         Fully-fetched AuthorizationResource objects.
     :ivar str fullchain_pem: The fetched contents of the certificate URL
         produced once the order was finalized, if it's present.
+    :ivar list of str alternative_fullchains_pem: The fetched contents of
+        alternative certificate chain URLs produced once the order was finalized,
+        if present and requested during finalization.
     """
     body = jose.Field('body', decoder=Order.from_json)
     csr_pem = jose.Field('csr_pem', omitempty=True)
     authorizations = jose.Field('authorizations')
     fullchain_pem = jose.Field('fullchain_pem', omitempty=True)
+    alternative_fullchains_pem = jose.Field('alternative_fullchains_pem', omitempty=True)
 
 @Directory.register
 class NewOrder(Order):

Mentioned in SAL (#wikimedia-operations) [2020-09-16T10:10:40Z] <ema> upload python-acme 0.31.0-2wm1 to buster-wikimedia T263006

Change 627809 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/software/acme-chief@master] x509: Provide support for an alternative chain

https://gerrit.wikimedia.org/r/627809

Change 627823 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/software/acme-chief@master] requests: Fetch alternative chains

https://gerrit.wikimedia.org/r/627823

Change 627869 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/software/acme-chief@master] api: Allow acme-chief clients to fetch alt. chain cert versions

https://gerrit.wikimedia.org/r/627869

Change 627873 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/software/acme-chief@master] Release 0.29

https://gerrit.wikimedia.org/r/627873

So I've prepared a 0.29 release shipping https://gerrit.wikimedia.org/r/q/topic:%22T263006%22+(status:open%20OR%20status:merged)

I've tested it manually on acmechief-test1001 and it works as expected:

root@acmechief-test1001:/var/lib/acme-chief/certs/apt/live# openssl x509 -serial -noout -in rsa-2048.chain.crt
serial=8BE12A0E5944ED3C546431F097614FE5
root@acmechief-test1001:/var/lib/acme-chief/certs/apt/live# openssl x509 -serial -noout -in rsa-2048.alt.chain.crt
serial=8BE12A0E5944ED3C546431F097614FE6
root@acmechief-test1001:/var/lib/acme-chief/certs/apt/live# ls -alh
total 96K
drwxr-x--- 2 acme-chief acme-chief 4.0K Sep 16 15:46 .
drwxr-xr-x 5 acme-chief acme-chief 4.0K Sep 16 15:45 ..
-rw-r--r-- 1 acme-chief acme-chief 1.4K Sep 16 15:45 ec-prime256v1.alt.chain.crt
-rw-r--r-- 1 acme-chief acme-chief 2.9K Sep 16 15:45 ec-prime256v1.alt.chained.crt
-rw-r----- 1 acme-chief acme-chief 3.1K Sep 16 15:45 ec-prime256v1.alt.chained.crt.key
-rw-r--r-- 1 acme-chief acme-chief 1.7K Sep 16 15:45 ec-prime256v1.chain.crt
-rw-r--r-- 1 acme-chief acme-chief 3.3K Sep 16 15:45 ec-prime256v1.chained.crt
-rw-r----- 1 acme-chief acme-chief 3.5K Sep 16 15:45 ec-prime256v1.chained.crt.key
-rw-r--r-- 1 acme-chief acme-chief 1.6K Sep 16 15:45 ec-prime256v1.crt
-rw-r----- 1 acme-chief acme-chief 1.8K Sep 16 15:45 ec-prime256v1.crt.key
-rw-r----- 1 acme-chief acme-chief  227 Sep 16 15:45 ec-prime256v1.key
-rw-r--r-- 1 acme-chief acme-chief  488 Sep 16 15:45 ec-prime256v1.ocsp
-rw-r--r-- 1 acme-chief acme-chief 1.4K Sep 16 15:45 rsa-2048.alt.chain.crt
-rw-r--r-- 1 acme-chief acme-chief 3.2K Sep 16 15:45 rsa-2048.alt.chained.crt
-rw-r----- 1 acme-chief acme-chief 4.8K Sep 16 15:45 rsa-2048.alt.chained.crt.key
-rw-r--r-- 1 acme-chief acme-chief 1.7K Sep 16 15:45 rsa-2048.chain.crt
-rw-r--r-- 1 acme-chief acme-chief 3.5K Sep 16 15:45 rsa-2048.chained.crt
-rw-r----- 1 acme-chief acme-chief 5.2K Sep 16 15:45 rsa-2048.chained.crt.key
-rw-r--r-- 1 acme-chief acme-chief 1.9K Sep 16 15:45 rsa-2048.crt
-rw-r----- 1 acme-chief acme-chief 3.5K Sep 16 15:45 rsa-2048.crt.key
-rw-r----- 1 acme-chief acme-chief 1.7K Sep 16 15:45 rsa-2048.key
-rw-r--r-- 1 acme-chief acme-chief  488 Sep 16 15:46 rsa-2048.ocsp

My current approach is just to release acme-chief 0.29, update the production environment and just wait till the next renewal cycle for the unified cert, cause it's due to be renewed in two days:

root@acmechief1001:/var/lib/acme-chief/certs/unified/live# openssl x509 -dates -noout -in rsa-2048.crt
notBefore=Jul 20 09:02:37 2020 GMT
notAfter=Oct 18 09:02:37 2020 GMT
root@acmechief1001:/var/lib/acme-chief/certs/unified/live# openssl x509 -dates -noout -in ec-prime256v1.crt
notBefore=Jul 20 09:02:07 2020 GMT
notAfter=Oct 18 09:02:07 2020 GMT

After that, we can switch on ats-tls to the alternative chain at will.

@BBlack are you ok with this approach?

Change 627809 merged by jenkins-bot:
[operations/software/acme-chief@master] x509: Alternative chain support

https://gerrit.wikimedia.org/r/627809

Change 627823 merged by jenkins-bot:
[operations/software/acme-chief@master] requests: Fetch alternative chains

https://gerrit.wikimedia.org/r/627823

Change 627869 merged by jenkins-bot:
[operations/software/acme-chief@master] api: Allow acme-chief clients to fetch alt. chain cert versions

https://gerrit.wikimedia.org/r/627869

Change 628053 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/software/acme-chief@debian] x509: Alternative chain support

https://gerrit.wikimedia.org/r/628053

Change 628054 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/software/acme-chief@debian] requests: Fetch alternative chains

https://gerrit.wikimedia.org/r/628054

Change 628056 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/software/acme-chief@debian] api: Allow acme-chief clients to fetch alt. chain cert versions

https://gerrit.wikimedia.org/r/628056

Change 627873 merged by jenkins-bot:
[operations/software/acme-chief@master] Release 0.29

https://gerrit.wikimedia.org/r/627873

Change 628057 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/software/acme-chief@debian] Release 0.29

https://gerrit.wikimedia.org/r/628057

Change 628067 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/software/acme-chief@debian] debian: Add release 0.29 to changelog

https://gerrit.wikimedia.org/r/628067

Change 628053 merged by jenkins-bot:
[operations/software/acme-chief@debian] x509: Alternative chain support

https://gerrit.wikimedia.org/r/628053

Change 628054 merged by jenkins-bot:
[operations/software/acme-chief@debian] requests: Fetch alternative chains

https://gerrit.wikimedia.org/r/628054

Change 628056 merged by jenkins-bot:
[operations/software/acme-chief@debian] api: Allow acme-chief clients to fetch alt. chain cert versions

https://gerrit.wikimedia.org/r/628056

Change 628057 merged by jenkins-bot:
[operations/software/acme-chief@debian] Release 0.29

https://gerrit.wikimedia.org/r/628057

Change 628067 merged by jenkins-bot:
[operations/software/acme-chief@debian] debian: Add release 0.29 to changelog

https://gerrit.wikimedia.org/r/628067

Mentioned in SAL (#wikimedia-operations) [2020-09-17T11:04:25Z] <vgutierrez> upload acme-chief 0.29 to apt.wm.o (buster) - T263006

Mentioned in SAL (#wikimedia-operations) [2020-09-17T11:06:12Z] <vgutierrez> update to acme-chief 0.29 on acmechief[12]001 - T263006

Vgutierrez lowered the priority of this task from High to Medium.Sep 17 2020, 11:11 AM

acme-chief updated to version 0.29 in our production environment, the unified cert should be renewed tomorrow, we will check the offered chains then

For the record, the renewal of unified on the 18th did successfully fetch two different chains, but they're currently set up with the default chain to the DST root, and the alternate chain to the ISRG root. This means we have nothing to do between now and the 29th for the unified, it will work as-is.

I'm guessing that the two will switch roles (alt to DST, default to ISRG) on or about the 29th for renewals from that point forward, in which case we can switch the unified and any others we care about deep compatibility for (wikiworkshop perhaps?) to "alt" in our config, later, on their next renwals. It's not ideal, but with only a couple of certs to focus on we can manage it. I suspect their intended workflow was that ACME clients would choose from the two intermediates based on a configured root name preference instead of caring which was in the alt slot explicitly, but this will work for us!