Page MenuHomePhabricator

curl on deployment-prep Buster hosts does not trust Puppet CA
Closed, ResolvedPublic

Description

deployment-etcd02 uses Puppet certs to provide etcd over https. curl can use it on Jessie hosts but not on Buster, and wget works just fine on Stretch:

taavi@deployment-mediawiki10:~$ curl-config --ca

taavi@deployment-mediawiki10:~$ curl https://deployment-etcd02.deployment-prep.eqiad1.wikimedia.cloud:2379
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
taavi@deployment-mediawiki10:~$ wget https://deployment-etcd02.deployment-prep.eqiad1.wikimedia.cloud:2379
--2021-03-29 09:12:18--  https://deployment-etcd02.deployment-prep.eqiad1.wikimedia.cloud:2379/
Resolving deployment-etcd02.deployment-prep.eqiad1.wikimedia.cloud (deployment-etcd02.deployment-prep.eqiad1.wikimedia.cloud)... 172.16.1.253
Connecting to deployment-etcd02.deployment-prep.eqiad1.wikimedia.cloud (deployment-etcd02.deployment-prep.eqiad1.wikimedia.cloud)|172.16.1.253|:2379... connected.
HTTP request sent, awaiting response... 404 Not Found
2021-03-29 09:12:18 ERROR 404: Not Found.

taavi@deployment-mediawiki-07:~$ curl-config --ca
/etc/ssl/certs/ca-certificates.crt
taavi@deployment-mediawiki-07:~$ curl https://deployment-etcd02.deployment-prep.eqiad1.wikimedia.cloud:2379
404 page not found

deployment-mediawiki-07 is on Stretch, while 10 is on Buster. The issue is also present on other Buster hosts.

On both OSes /etc/ssl/certs/ca-certificates.crt and /etc/ssl/certs/ca-certificates.crt both contain the ca. strace shows that Buster does not read any CA files:

taavi@deployment-mediawiki10:~$ strace curl https://deployment-etcd02.deployment-prep.eqiad1.wikimedia.cloud:2379 |& grep open | grep ssl
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libssl.so.1.1", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/usr/lib/ssl/openssl.cnf", O_RDONLY) = 3
openat(AT_FDCWD, "/usr/lib/ssl/openssl.cnf", O_RDONLY) = 4

taavi@deployment-mediawiki-07:~$ strace curl https://deployment-etcd02.deployment-prep.eqiad1.wikimedia.cloud:2379 |& grep open | grep ssl
open("/usr/lib/x86_64-linux-gnu/libssl.so.1.0.2", O_RDONLY|O_CLOEXEC) = 3
open("/usr/lib/ssl/openssl.cnf", O_RDONLY) = 3
open("/etc/ssl/certs/ca-certificates.crt", O_RDONLY) = 4

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald Transcript
taavi claimed this task.

yep, it was, thanks! the following one-liner solved it:

taavi@deployment-mediawiki11:~$ sudo ln -s /usr/local/share/ca-certificates/Puppet_Internal_CA.crt /etc/ssl/certs/$(openssl x509 -noout -hash -in /usr/local/share/ca-certificates/Puppet_Internal_CA.crt).0 && sudo update-ca-certificates

It would be nice to document this someplace so that the next time someone runs into the issue they have a hope of finding the answer on wikitech instead of opening another task. But I don't know where that someplace is...