Page MenuHomePhabricator

/etc/ssl/certs/Puppet_Internal_CA.pem overridden to production certs on cloud vps
Closed, ResolvedPublic


As a part of today's unattended-upgrades run, the wmf-certificates package was automatically upgraded causing the /etc/ssl/certs/Puppet_Internal_CA.pem file to be overridden with production values.

Puppet itself uses /var/lib/puppet/ssl/certs/ca.pem (untouched), but this has broken several other things, at least T296125: Fatal error: Uncaught ConfigException: Failed to load configuration from etcd

Event Timeline

taavi triaged this task as Unbreak Now! priority.Nov 20 2021, 9:58 AM
taavi created this task.
elukey@deployment-mediawiki12:~$ ls -l /etc/ssl/certs/Puppet_Internal_CA.pem 
lrwxrwxrwx 1 root root 59 Nov 20 06:24 /etc/ssl/certs/Puppet_Internal_CA.pem -> /usr/share/ca-certificates/wikimedia/Puppet_Internal_CA.crt

Report from @Majavah on IRC:

<majavah> 679 instances with the wrong production one, 136 in toolforge with the correct one, then a few with other project local CAs
<majavah> taavi@cloud-cumin-03:~$ sudo cumin "A:all" "openssl x509 -in /etc/ssl/certs/Puppet_Internal_CA.pem -text -noout | grep CN"

@elukey Can you tell me what you mean by 'last change'? I can't tell if this patch is meant to resolve the issue, or forestall future similar issues, or what :)

elukey@deployment-webperf11:~$ ls -l /usr/local/share/ca-certificates | grep Puppet
-r--r--r-- 1 root root 2122 Jun  1  2018 Puppet_Internal_CA.crt

The change in was meant for production, to create a bundle of Puppet CA crt + Root PKI CA crt via update-ca-certificates without adding to it all the certs under /usr/local/share/ca-certificates. It worked nicely in production, but afaics in cloud caused the Puppet CA cert under /usr/local/share to be ignored and the one created under /usr/share/ca-certificates to be picked instead (containing the production Puppet CA crt installed by the package).

The problem is still to be resolved :)

Change 740389 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] profile::base::certificates: deploy wmf-certificates only in prod

Change 740547 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/debs/wmf-certificates@main] Reimplement hook to no longer call update-ca-certificates

The hook to update wmf-ca-certificates.crt does call update-ca-certificates again with a different configuration including ignoring certificates in /usr/local/share/ca-certificates as that led to having duplicate (and non-wmf) certificates in the bundle in production. That led update-ca-certificates to link the Puppet-CA included in the package rather than the /usr/local/share/ one to /etc/ssl/certs.

With Change 740547 I've reimplemented the hook to no longer use using update-ca-certificates to create the wmf-ca-certificates bundle which should restore old behavior.

Change 740547 merged by JMeybohm:

[operations/debs/wmf-certificates@main] Reimplement hook to no longer call update-ca-certificates

I've just imported wmf-certificates version 0~20211122-1 that should fix the situation

taavi claimed this task.
taavi reassigned this task from taavi to JMeybohm.

Change 740389 merged by Elukey:

[operations/puppet@production] profile::base::certificates: deploy wmf-certificates only in prod

To keep archives happy - wmf-certificates is now deployed only in production, not in deployment-prep.