Puppet itself uses /var/lib/puppet/ssl/certs/ca.pem (untouched), but this has broken several other things, at least T296125: Fatal error: Uncaught ConfigException: Failed to load configuration from etcd
|Resolved||Majavah||T296125 Fatal error: Uncaught ConfigException: Failed to load configuration from etcd|
|Resolved||JMeybohm||T296127 /etc/ssl/certs/Puppet_Internal_CA.pem overridden to production certs on cloud vps|
elukey@deployment-mediawiki12:~$ ls -l /etc/ssl/certs/Puppet_Internal_CA.pem lrwxrwxrwx 1 root root 59 Nov 20 06:24 /etc/ssl/certs/Puppet_Internal_CA.pem -> /usr/share/ca-certificates/wikimedia/Puppet_Internal_CA.crt
Report from @Majavah on IRC:
<majavah> 679 instances with the wrong production one, 136 in toolforge with the correct one, then a few with other project local CAs <majavah> taavi@cloud-cumin-03:~$ sudo cumin "A:all" "openssl x509 -in /etc/ssl/certs/Puppet_Internal_CA.pem -text -noout | grep CN"
elukey@deployment-webperf11:~$ ls -l /usr/local/share/ca-certificates | grep Puppet -r--r--r-- 1 root root 2122 Jun 1 2018 Puppet_Internal_CA.crt
The change in https://gerrit.wikimedia.org/r/c/operations/debs/wmf-certificates/+/740119 was meant for production, to create a bundle of Puppet CA crt + Root PKI CA crt via update-ca-certificates without adding to it all the certs under /usr/local/share/ca-certificates. It worked nicely in production, but afaics in cloud caused the Puppet CA cert under /usr/local/share to be ignored and the one created under /usr/share/ca-certificates to be picked instead (containing the production Puppet CA crt installed by the package).
The problem is still to be resolved :)
The hook to update wmf-ca-certificates.crt does call update-ca-certificates again with a different configuration including ignoring certificates in /usr/local/share/ca-certificates as that led to having duplicate (and non-wmf) certificates in the bundle in production. That led update-ca-certificates to link the Puppet-CA included in the package rather than the /usr/local/share/ one to /etc/ssl/certs.
With Change 740547 I've reimplemented the hook to no longer use using update-ca-certificates to create the wmf-ca-certificates bundle which should restore old behavior.