As a part of today's unattended-upgrades run, the wmf-certificates package was automatically upgraded causing the /etc/ssl/certs/Puppet_Internal_CA.pem file to be overridden with production values.
Puppet itself uses /var/lib/puppet/ssl/certs/ca.pem (untouched), but this has broken several other things, at least T296125: Fatal error: Uncaught ConfigException: Failed to load configuration from etcd
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | • taavi | T296125 Fatal error: Uncaught ConfigException: Failed to load configuration from etcd | |||
| Resolved | JMeybohm | T296127 /etc/ssl/certs/Puppet_Internal_CA.pem overridden to production certs on cloud vps |
elukey@deployment-mediawiki12:~$ ls -l /etc/ssl/certs/Puppet_Internal_CA.pem lrwxrwxrwx 1 root root 59 Nov 20 06:24 /etc/ssl/certs/Puppet_Internal_CA.pem -> /usr/share/ca-certificates/wikimedia/Puppet_Internal_CA.crt
Report from @Majavah on IRC:
<majavah> 679 instances with the wrong production one, 136 in toolforge with the correct one, then a few with other project local CAs <majavah> taavi@cloud-cumin-03:~$ sudo cumin "A:all" "openssl x509 -in /etc/ssl/certs/Puppet_Internal_CA.pem -text -noout | grep CN"
Last change for wmf-certificates https://gerrit.wikimedia.org/r/c/operations/debs/wmf-certificates/+/740119
@elukey Can you tell me what you mean by 'last change'? I can't tell if this patch is meant to resolve the issue, or forestall future similar issues, or what :)
elukey@deployment-webperf11:~$ ls -l /usr/local/share/ca-certificates | grep Puppet -r--r--r-- 1 root root 2122 Jun 1 2018 Puppet_Internal_CA.crt
The change in https://gerrit.wikimedia.org/r/c/operations/debs/wmf-certificates/+/740119 was meant for production, to create a bundle of Puppet CA crt + Root PKI CA crt via update-ca-certificates without adding to it all the certs under /usr/local/share/ca-certificates. It worked nicely in production, but afaics in cloud caused the Puppet CA cert under /usr/local/share to be ignored and the one created under /usr/share/ca-certificates to be picked instead (containing the production Puppet CA crt installed by the package).
The problem is still to be resolved :)
Change 740389 had a related patch set uploaded (by Elukey; author: Elukey):
[operations/puppet@production] profile::base::certificates: deploy wmf-certificates only in prod
Change 740547 had a related patch set uploaded (by JMeybohm; author: JMeybohm):
[operations/debs/wmf-certificates@main] Reimplement hook to no longer call update-ca-certificates
The hook to update wmf-ca-certificates.crt does call update-ca-certificates again with a different configuration including ignoring certificates in /usr/local/share/ca-certificates as that led to having duplicate (and non-wmf) certificates in the bundle in production. That led update-ca-certificates to link the Puppet-CA included in the package rather than the /usr/local/share/ one to /etc/ssl/certs.
With Change 740547 I've reimplemented the hook to no longer use using update-ca-certificates to create the wmf-ca-certificates bundle which should restore old behavior.
Change 740547 merged by JMeybohm:
[operations/debs/wmf-certificates@main] Reimplement hook to no longer call update-ca-certificates
I've just imported wmf-certificates version 0~20211122-1 that should fix the situation
Change 740389 merged by Elukey:
[operations/puppet@production] profile::base::certificates: deploy wmf-certificates only in prod
To keep archives happy - wmf-certificates is now deployed only in production, not in deployment-prep.