Page MenuHomePhabricator

OpenSSL < 1.1.0 compatibility issues with new LE issuance chain
Closed, ResolvedPublic

Description

As detailed in https://community.letsencrypt.org/t/openssl-client-compatibility-changes-for-let-s-encrypt-certificates/143816, the new Let's Encrypt default issuance chain ISRG Root X1 <-- DST Root CA X3 triggers a certificate chain verification corner case that affects the following libraries:

  • OpenSSL < 1.1.0
  • LibreSSL < 3.2.0
  • GnuTLS < 3.6.14

This issue as mentioned on the GnuTLS ticket affected AddTrust CA almost one year ago and the patch for the main libraries has been probably backported on the major linux distributions.

Related Objects

Event Timeline

As mentioned on the issue description, debian backported the fix for OpenSSL as it can be seen on a current debian jessie container:

root@69310d82543d:~# cat /etc/debian_version 
8.11
root@69310d82543d:~# openssl version
OpenSSL 1.0.1t  3 May 2016
root@69310d82543d:~# openssl verify -CAfile rsa-2048.chain.crt rsa-2048.crt 
rsa-2048.crt: OK
root@69310d82543d:~# openssl x509 -dates -noout -in rsa-2048.crt 
notBefore=May 10 13:15:07 2021 GMT
notAfter=Aug  8 13:15:07 2021 GMT

As mentioned on the issue description, debian backported the fix for OpenSSL as it can be seen on a current debian jessie container:

root@69310d82543d:~# cat /etc/debian_version 
8.11
root@69310d82543d:~# openssl version
OpenSSL 1.0.1t  3 May 2016
root@69310d82543d:~# openssl verify -CAfile rsa-2048.chain.crt rsa-2048.crt 
rsa-2048.crt: OK
root@69310d82543d:~# openssl x509 -dates -noout -in rsa-2048.crt 
notBefore=May 10 13:15:07 2021 GMT
notAfter=Aug  8 13:15:07 2021 GMT

Stretch has two copies of OpenSSL; src:openssl ships 1.1.0l (recent enough), but also src:openssl1.0 for packages which weren't ported to the new OpenSSL 1.1 API back then. This package is affected, but we need a closer look at which applications use it. Some are self-contained (e.g. OpenSSH), but e.g. Ruby and Node.js on Stretch still use OpenSSL 1.0.2.

Likewise for GNU TLS on Stretch. It's less used compared to OpenSSL, but e.g. still affects the GNUTLS-based flavours of libcurl.

akosiaris triaged this task as Medium priority.Sep 10 2021, 2:38 PM

Mentioned in SAL (#wikimedia-operations) [2021-09-20T09:10:15Z] <moritzm> installing openssl1.0 updates for stretch with backport for forthcoming Let's encrypt issuance chain update (T283165)

Mentioned in SAL (#wikimedia-operations) [2021-09-20T12:54:10Z] <moritzm> installing gnutls28 updates for stretch with backport for forthcoming Let's encrypt issuance chain update (T283165)

Mentioned in SAL (#wikimedia-operations) [2021-09-20T13:45:04Z] <moritzm> restarting apache on Logstash ELK5 cluster to pick up GNUTLS update T283165

For production:

  • OpenSSL in Buster and Bullseye is not affected (only ship OpenSSL 1.1)
  • OpenSSL updates for openssl 1.0.2 in Stretch have been rolled out
  • GNUTLS in Bullseye is not affected
  • GNUTLS in Buster was already fixed in Buster 10.10 (rolled out via T285206)
  • GNUTLS updates for Stretch have been rolled out

A few container images might need to be rebuilt for production and CI (not sure which of these are still in use):
https://debmonitor.wikimedia.org/packages/libssl1.0.2

Change 720241 had a related patch set uploaded (by Alexandros Kosiaris; author: Hashar):

[operations/puppet@production] docker: add security updates to Bullseye base image

https://gerrit.wikimedia.org/r/720241

Change 720241 had a related patch set uploaded (by Alexandros Kosiaris; author: Hashar):

[operations/puppet@production] docker: add security updates to Bullseye base image

https://gerrit.wikimedia.org/r/720241

Wrong bug added, ignore please.

For production:

  • OpenSSL in Buster and Bullseye is not affected (only ship OpenSSL 1.1)
  • OpenSSL updates for openssl 1.0.2 in Stretch have been rolled out
  • GNUTLS in Bullseye is not affected
  • GNUTLS in Buster was already fixed in Buster 10.10 (rolled out via T285206)
  • GNUTLS updates for Stretch have been rolled out

A few container images might need to be rebuilt for production and CI (not sure which of these are still in use):
https://debmonitor.wikimedia.org/packages/libssl1.0.2

With T291458 done, I 've already rebuilt bullseye (which was not affected) and buster main images (with libgnutls30 3.6.7-4+deb10u7) so I think the base layers are done.

I 'll delete docker-registry.wikimedia.org/wikimedia/mediawiki-services-graphoid:2019-06-10-060747-production as graphoid is no longer around

I 'll also rebuild

  • docker-registry.wikimedia.org/nodejs-slim:0.0.2-20210912
  • docker-registry.wikimedia.org/ruby:0.0.2-20210912

For the releng images, I guess release engineering should be pinged.

For production:

  • OpenSSL in Buster and Bullseye is not affected (only ship OpenSSL 1.1)
  • OpenSSL updates for openssl 1.0.2 in Stretch have been rolled out
  • GNUTLS in Bullseye is not affected
  • GNUTLS in Buster was already fixed in Buster 10.10 (rolled out via T285206)
  • GNUTLS updates for Stretch have been rolled out

A few container images might need to be rebuilt for production and CI (not sure which of these are still in use):
https://debmonitor.wikimedia.org/packages/libssl1.0.2

With T291458 done, I 've already rebuilt bullseye (which was not affected) and buster main images (with libgnutls30 3.6.7-4+deb10u7) so I think the base layers are done.

I 'll delete docker-registry.wikimedia.org/wikimedia/mediawiki-services-graphoid:2019-06-10-060747-production as graphoid is no longer around

I 'll also rebuild

  • docker-registry.wikimedia.org/nodejs-slim:0.0.2-20210912
  • docker-registry.wikimedia.org/ruby:0.0.2-20210912

For the releng images, I guess release engineering should be pinged.

Heh, actually done already in https://phabricator.wikimedia.org/T291458#7368374

I missed that.

With T291458 done, I 've already rebuilt bullseye (which was not affected) and buster main images (with libgnutls30 3.6.7-4+deb10u7) so I think the base layers are done.

I 'll delete docker-registry.wikimedia.org/wikimedia/mediawiki-services-graphoid:2019-06-10-060747-production as graphoid is no longer around

I 'll also rebuild

  • docker-registry.wikimedia.org/nodejs-slim:0.0.2-20210912
  • docker-registry.wikimedia.org/ruby:0.0.2-20210912

Ack.

For the releng images, I guess release engineering should be pinged.

That was completed via https://phabricator.wikimedia.org/T291425