Page MenuHomePhabricator

Consider ways to make puppetmaster CA changes smoother on the puppet client end
Open, Needs TriagePublic

Description

I've been maintaining instances in the labs realm with custom puppetmasters for a while, and when setting up such instances the steps needed to make the client able to talk to the new puppetmaster (that it's just pulled into puppet.conf from hiera) are this:
cd /var/lib/puppet; mv ssl ssl_old; rm /usr/local/share/ca-certificates/Puppet_Internal_CA.crt; nano /usr/local/share/ca-certificates/Puppet_Internal_CA.crt; update-ca-certificates --fresh; puppet agent -tv
(pasting the contents of the file in from an existing client with it, or the puppetmaster's copy of the cert)
Alternatively, something like this as I've been doing for T171188 test clients:
cd /var/lib/puppet; mv ssl ssl.$(date '+%Y-%m-%dT%H:%M'); curl https://phab.wmfusercontent.org/file/data/sp3m7a6mjr53xfwlidz7/PHID-FILE-s4vhserqjh34z764hk6s/raw.txt -o /usr/local/share/ca-certificates/Puppet_Internal_CA.crt -s; update-ca-certificates --fresh; puppet agent -tv
(obviously only using a URL you trust for this, that particular one is a paste I made in phab)
If the new puppetmaster is autosigning then this is all. If not then obviously the user will need to get the cert signed, but that's it.

These commands do two things:

  • Clear the /var/lib/puppet/ssl directory so old client certificates stop being used and new client certificates are generated instead.
  • Replace the /usr/local/share/ca-certificates/Puppet_Internal_CA.crt file and run the appropriate update command.

That Puppet_Internal_CA.crt file comes from the Sslcert::Ca['Puppet_Internal_CA'] resource in profile::base::certificates, with source "${puppet_ssl_dir}/certs/ca.pem" ($puppet_ssl_dir = puppet_ssldir()), so the old puppetmaster will leave the contents of the file being its own CA, and the puppet client will not pick up the one from the new puppetmaster as it will not yet trust it - hence the above commands.

But we could change this - maybe we could store all known puppetmaster's CA files in puppet.git, and make that source configurable in hiera. That way when changing the puppetmaster key we could also have the old puppetmaster instruct the client to also change the cert they trust (alongside the master config update in puppet.conf), making that change automatic. It should also exec the update-ca-certificates stuff.
Alternatively maybe the CA certificate contents could come from hiera.
We could also add an Exec['clear-old-puppet-ssl'], notified by this Sslcert::Ca resource, which takes care of the /var/lib/puppet/ssl move.

Thoughts? This probably isn't strictly required for the parent/grandparent tickets but may make them, and normal project puppetmaster operations, easier.

Event Timeline

Krenair created this task.Apr 6 2019, 1:59 PM
Krenair updated the task description. (Show Details)Apr 6 2019, 11:31 PM

I'm wary of having a central repo of alternate puppetmasters (mostly because maintaining it seems like a pain) but it's not out of the question.

Would it make sense to have the firstboot script (which already runs puppet several times) notice if the requested puppetmaster changes during initial setup and, if it does, wipe out existing certs and re-run?

Another option is to switch to using an actual generic 'puppet' name for the puppetmaster, and hack per-project dns to point to the correct puppetmaster from the outset. I guess that would break a lot of things for projects without autosigning though.

I'm wary of having a central repo of alternate puppetmasters (mostly because maintaining it seems like a pain) but it's not out of the question.

From a puppet.git point of view we don't really need to store any certs, just make them changeable via hiera.

Would it make sense to have the firstboot script (which already runs puppet several times) notice if the requested puppetmaster changes during initial setup and, if it does, wipe out existing certs and re-run?

Possibly but that would only help on firstboot, not changes made during the lifetime of an instance. It wouldn't actually help migrations.

Another option is to switch to using an actual generic 'puppet' name for the puppetmaster, and hack per-project dns to point to the correct puppetmaster from the outset. I guess that would break a lot of things for projects without autosigning though.

That wouldn't really solve anything as their CAs would all still be separate even if they were named the same.

Change 506872 had a related patch set uploaded (by Alex Monk; owner: Alex Monk):
[operations/puppet@production] Puppet CAs: Make it easy to swap CAs by hiera change

https://gerrit.wikimedia.org/r/506872

Change 506873 had a related patch set uploaded (by Alex Monk; owner: Alex Monk):
[operations/puppet@production] Puppet certs: Move old client certs away when Puppet CA changes

https://gerrit.wikimedia.org/r/506873