Page MenuHomePhabricator

Consider ways to make puppetmaster CA changes smoother on the puppet client end
Closed, ResolvedPublic

Description

I've been maintaining instances in the labs realm with custom puppetmasters for a while, and when setting up such instances the steps needed to make the client able to talk to the new puppetmaster (that it's just pulled into puppet.conf from hiera) are this:
cd /var/lib/puppet; mv ssl ssl_old; rm /usr/local/share/ca-certificates/Puppet_Internal_CA.crt; nano /usr/local/share/ca-certificates/Puppet_Internal_CA.crt; update-ca-certificates --fresh; puppet agent -tv
(pasting the contents of the file in from an existing client with it, or the puppetmaster's copy of the cert)
Alternatively, something like this as I've been doing for T171188 test clients:
cd /var/lib/puppet; mv ssl ssl.$(date '+%Y-%m-%dT%H:%M'); curl https://phab.wmfusercontent.org/file/data/sp3m7a6mjr53xfwlidz7/PHID-FILE-s4vhserqjh34z764hk6s/raw.txt -o /usr/local/share/ca-certificates/Puppet_Internal_CA.crt -s; update-ca-certificates --fresh; puppet agent -tv
(obviously only using a URL you trust for this, that particular one is a paste I made in phab)
If the new puppetmaster is autosigning then this is all. If not then obviously the user will need to get the cert signed, but that's it.

These commands do two things:

  • Clear the /var/lib/puppet/ssl directory so old client certificates stop being used and new client certificates are generated instead.
  • Replace the /usr/local/share/ca-certificates/Puppet_Internal_CA.crt file and run the appropriate update command.

That Puppet_Internal_CA.crt file comes from the Sslcert::Ca['Puppet_Internal_CA'] resource in profile::base::certificates, with source "${puppet_ssl_dir}/certs/ca.pem" ($puppet_ssl_dir = puppet_ssldir()), so the old puppetmaster will leave the contents of the file being its own CA, and the puppet client will not pick up the one from the new puppetmaster as it will not yet trust it - hence the above commands.

But we could change this - maybe we could store all known puppetmaster's CA files in puppet.git, and make that source configurable in hiera. That way when changing the puppetmaster key we could also have the old puppetmaster instruct the client to also change the cert they trust (alongside the master config update in puppet.conf), making that change automatic. It should also exec the update-ca-certificates stuff.
Alternatively maybe the CA certificate contents could come from hiera.
We could also add an Exec['clear-old-puppet-ssl'], notified by this Sslcert::Ca resource, which takes care of the /var/lib/puppet/ssl move.

Thoughts? This probably isn't strictly required for the parent/grandparent tickets but may make them, and normal project puppetmaster operations, easier.

Event Timeline

I'm wary of having a central repo of alternate puppetmasters (mostly because maintaining it seems like a pain) but it's not out of the question.

Would it make sense to have the firstboot script (which already runs puppet several times) notice if the requested puppetmaster changes during initial setup and, if it does, wipe out existing certs and re-run?

Another option is to switch to using an actual generic 'puppet' name for the puppetmaster, and hack per-project dns to point to the correct puppetmaster from the outset. I guess that would break a lot of things for projects without autosigning though.

I'm wary of having a central repo of alternate puppetmasters (mostly because maintaining it seems like a pain) but it's not out of the question.

From a puppet.git point of view we don't really need to store any certs, just make them changeable via hiera.

Would it make sense to have the firstboot script (which already runs puppet several times) notice if the requested puppetmaster changes during initial setup and, if it does, wipe out existing certs and re-run?

Possibly but that would only help on firstboot, not changes made during the lifetime of an instance. It wouldn't actually help migrations.

Another option is to switch to using an actual generic 'puppet' name for the puppetmaster, and hack per-project dns to point to the correct puppetmaster from the outset. I guess that would break a lot of things for projects without autosigning though.

That wouldn't really solve anything as their CAs would all still be separate even if they were named the same.

Change 506872 had a related patch set uploaded (by Alex Monk; owner: Alex Monk):
[operations/puppet@production] Puppet CAs: Make it easy to swap CAs by hiera change

https://gerrit.wikimedia.org/r/506872

Change 506873 had a related patch set uploaded (by Alex Monk; owner: Alex Monk):
[operations/puppet@production] Puppet certs: Move old client certs away when Puppet CA changes

https://gerrit.wikimedia.org/r/506873

Change 506872 merged by Andrew Bogott:
[operations/puppet@production] Puppet CAs: Make it easy to swap CAs by hiera change

https://gerrit.wikimedia.org/r/506872

Change 506873 merged by Andrew Bogott:
[operations/puppet@production] Puppet certs: Move old client certs away when Puppet CA changes

https://gerrit.wikimedia.org/r/506873

Krenair closed this task as Resolved.EditedJun 22 2019, 4:31 PM
Krenair claimed this task.

Wish I'd done this years ago. It seems to have worked and allows us to effortlessly move instances between puppetmasters - which should prove very helpful to the puppetmaster realm migration.

I made https://wikitech.wikimedia.org/w/index.php?title=Hiera:Deployment-prep&diff=1830131&oldid=1824364 which just imports the deployment-puppetmaster03 CA into hiera (it's on wikitech because it's kind of the logical place to find it given the puppetmaster var is set just above in there), I then made an instance in deployment-prep, and it works:

krenair@deployment-alex-test:/$ sudo puppet agent -tv
Exiting; no certificate found and waitforcert is disabled

hopped on the puppetmaster and signed it, it all worked:

krenair@deployment-alex-test:/$ sudo puppet agent -tv
Info: Caching certificate for deployment-alex-test.deployment-prep.eqiad.wmflabs

This hopefully renders obsolete the silly puppet bootstrapping dance performed on every instance creation in that project, and any others that set puppetmaster on a project/prefix basis. They will just need to set profile::base::certificates::puppet_ca_content appropriately where they already set puppetmaster.
There was one thing broken about this new instance that appeared to sort itself out, haven't dug into it yet. Most likely not puppet related as puppet appears quite happy.
Edit: Dug into it - it being my home directory not being created on login - believe it was the lack of a /usr/share/pam-configs/wikimedia-labs-pam file, which puppet is responsible for creating. Not sure why the initial puppet run did not take care of that. Still, quite a minor thing, and it is solved as soon as the instance gets its puppet cert signed and agent run.


The next thing I did was for the parent ticket - I set following hiera data on the specific instance:

puppetmaster: puppetmaster.cloudinfra.wmflabs.org
"profile::base::certificates::puppet_ca_content": |
  -----BEGIN CERTIFICATE-----
  MIIF/zCCA+egAwIBAgIBATANBgkqhkiG9w0BAQsFADBRMU8wTQYDVQQDDEZQdXBw
  ZXQgQ0E6IGNsb3VkaW5mcmEtaW50ZXJuYWwtcHVwcGV0bWFzdGVyMDEuY2xvdWRp
  bmZyYS5lcWlhZC53bWZsYWJzMB4XDTE5MDQwMTIwMzUxMFoXDTI0MDMzMTIwMzUx
  MFowUTFPME0GA1UEAwxGUHVwcGV0IENBOiBjbG91ZGluZnJhLWludGVybmFsLXB1
  cHBldG1hc3RlcjAxLmNsb3VkaW5mcmEuZXFpYWQud21mbGFiczCCAiIwDQYJKoZI
  hvcNAQEBBQADggIPADCCAgoCggIBAMZLCbq/E9ZlNKqOz45I/qbZzJiTzDQ8yNge
  N8UnaJ2d56D9PkYNQ4H3DOCdcXj7fWB8CHF6yC0CAI43XrDLvUEfuFfpLn4TxmFY
  AypgrEmvYW62rfsoZe1/6qzo+CJfPpE+OIPZlpk1OzKMjRFIJOgOFu0Sp/kgisPK
  Lx/7qiapWB7+NGLpUCzJeMllcRWbzZZo2vuh9hh/XUWLmb8Z/t+4zPsewIHlpYNE
  f9y0fPDPjGbR3eA+Aif1XjMKRcN8KCpbUBWFh8kO4ivnXfxfCBgHeLtZDUwOJEDW
  tSTzfQZQ4FOYds8l2ruB1fqkTNSBjfnILd0gB7c/QgRn1vQ7KzN5YJfcVgBZSTk2
  JehmQKHgZtCn3hjuZ3RoN9+o9IuQCprePs8frivLZqbn6GuPufnUhT4GMdjKMQYc
  2p9IJ9TyExcgst0KoXc3H/JUz24PgCiwprxpBGxEqsDBfcCm0iUMOHVPIJvla+Em
  uEW12kXdeKbQHWMDUDWQfAdOJVlL0faaw3qHA5W8hM8kS4cVmna50BI9DqbmCr3e
  zekiib3MbNSi9NNlTf/ZCAkbrCIenvwY6Aok+2jJ2+600TsrDr3ncWuZ61cN6d6W
  tA06i7LMfJRFpP1TrddFYuZVn6NJ7hwrLq4FU34OKZD8aSTs8ZSDhgmAkY+aE0t7
  J2oFxACDAgMBAAGjgeEwgd4wNwYJYIZIAYb4QgENBCoMKFB1cHBldCBSdWJ5L09w
  ZW5TU0wgSW50ZXJuYWwgQ2VydGlmaWNhdGUwDgYDVR0PAQH/BAQDAgEGMA8GA1Ud
  EwEB/wQFMAMBAf8wHQYDVR0OBBYEFBVB8b3TIGyi+0i2Db19AcgE0rwFMGMGA1Ud
  IwRcMFqhVaRTMFExTzBNBgNVBAMMRlB1cHBldCBDQTogY2xvdWRpbmZyYS1pbnRl
  cm5hbC1wdXBwZXRtYXN0ZXIwMS5jbG91ZGluZnJhLmVxaWFkLndtZmxhYnOCAQEw
  DQYJKoZIhvcNAQELBQADggIBAEdQHweh1Ff2l/m8PI5SbYLLFv7wNJTwokNt+x/h
  2uABgXxhZsl9zzacaM66XiTb7o9/ZyJRz/YXMvI0EFYcWfWyvttXJUO02BFnpY1n
  aGEH+rqCbY+3PkiSLrU3rLNxw8XLci5ggR/B6e1FZTR2X1gln1Lpr8mZ9Ph7F9dq
  8jcV6im8kzzi4qTTOqzbMJuiDlQhAK7puNi65O1mPs0MJB0/NXCNV/0/xWD6Bcyv
  I0+aJk3CJcif6LWmv2XpX/GudFT7NdY5ha3nCcRyu7ofKshQ3gPrbnyj0CnYO17P
  Arb/yHEaYDyz8P06/MiTZ/tiBo1FrE1nfdcxjIvasuBQuY11GJyXcuBKcStXyEO9
  UPioqXA8Sxi50yNe3s4JaZiwFbc18VD05CKrlF5aVqGONH1tzlnhP9lu7R1DeOs4
  2RyXkhW/5/xDeWaVaeCOWRCn5RAKmyBFm4gJuIxWxg7hDk1RbtyaYvJgGRzUQVJs
  OvnfgkbwoAGACAnT+p4S2k7dhqC+r9q51JAMiR0CPL+EO+OGtw2j61bKXxjU4tmp
  FO/uxPPgZfPjDs3/YjhEZatl2cAX0FcU6VoSZT25VtRS+4bZHOVzpmb0q2WquApU
  sOaizh9f/VqmkVXlZ/xyFDk41wThP6aNG1U3ghHmcfTW/95YRdddfb5zNZUnf0EQ
  8x2v
  -----END CERTIFICATE-----

(cert is from cloud-puppetmaster-01.cloudinfra.eqiad.wmflabs)

Next few puppet runs worked like you'd expect:

root@deployment-alex-test:/var/lib/puppet# puppet agent -tv
Info: Using configured environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Info: Caching catalog for deployment-alex-test.deployment-prep.eqiad.wmflabs
Notice: /Stage[main]/Base::Environment/Tidy[/var/tmp/core]: Tidying 0 files
Info: Applying configuration version '1561220515'
Notice: openstack::clientpackages::vms::mitaka::stretch: no special configuration yet
Notice: /Stage[main]/Openstack::Clientpackages::Vms::Mitaka::Stretch/Notify[openstack::clientpackages::vms::mitaka::stretch: no special configuration yet]/message: defined 'message' as 'openstack::clientpackages::vms::mitaka::stretch: no special configuration yet'
Notice: The LDAP client stack for this host is: classic/sudoldap
Notice: /Stage[main]/Profile::Ldap::Client::Labs/Notify[LDAP client stack]/message: defined 'message' as 'The LDAP client stack for this host is: classic/sudoldap'
Notice: /Stage[main]/Profile::Base::Certificates/Sslcert::Ca[Puppet_Internal_CA]/File[/usr/local/share/ca-certificates/Puppet_Internal_CA.crt]/content: 
--- /usr/local/share/ca-certificates/Puppet_Internal_CA.crt	2019-06-22 15:19:56.752206034 +0000
+++ /tmp/puppet-file20190622-1173-19iyfwa	2019-06-22 16:22:04.061375536 +0000
[nb: snip cert diff from paste]

Info: Computing checksum on file /usr/local/share/ca-certificates/Puppet_Internal_CA.crt
Info: FileBucket got a duplicate file {md5}7b1e67a47e939c2103e723fbb48712d7
Info: /Stage[main]/Profile::Base::Certificates/Sslcert::Ca[Puppet_Internal_CA]/File[/usr/local/share/ca-certificates/Puppet_Internal_CA.crt]: Filebucketed /usr/local/share/ca-certificates/Puppet_Internal_CA.crt to puppet with sum 7b1e67a47e939c2103e723fbb48712d7
Notice: /Stage[main]/Profile::Base::Certificates/Sslcert::Ca[Puppet_Internal_CA]/File[/usr/local/share/ca-certificates/Puppet_Internal_CA.crt]/content: content changed '{md5}7b1e67a47e939c2103e723fbb48712d7' to '{md5}88238be8621899537b65bcb35527be2e'
Info: /Stage[main]/Profile::Base::Certificates/Sslcert::Ca[Puppet_Internal_CA]/File[/usr/local/share/ca-certificates/Puppet_Internal_CA.crt]: Scheduling refresh of Exec[update-ca-certificates]
Notice: /Stage[main]/Sslcert/Exec[update-ca-certificates]: Triggered 'refresh' from 1 events
Info: Sslcert::Ca[Puppet_Internal_CA]: Scheduling refresh of Exec[clear-old-puppet-ssl]
Notice: /Stage[main]/Profile::Base::Certificates/Exec[clear-old-puppet-ssl]: Triggered 'refresh' from 1 events
Notice: /Stage[main]/Base::Puppet/Base::Puppet::Config[main]/File[/etc/puppet/puppet.conf.d/10-main.conf]/content: 
--- /etc/puppet/puppet.conf.d/10-main.conf	2019-06-22 15:21:14.729715428 +0000
+++ /tmp/puppet-file20190622-1173-mlkemu	2019-06-22 16:22:05.229399018 +0000
@@ -11,7 +11,7 @@
 factpath = $vardir/lib/facter
 
 [agent]
-server = deployment-puppetmaster03.deployment-prep.eqiad.wmflabs
+server = puppetmaster.cloudinfra.wmflabs.org
 
 
 daemonize = false

Info: Computing checksum on file /etc/puppet/puppet.conf.d/10-main.conf
Info: FileBucket got a duplicate file {md5}73ac1d853f8c1aeb622e53a03efe0b07
Info: /Stage[main]/Base::Puppet/Base::Puppet::Config[main]/File[/etc/puppet/puppet.conf.d/10-main.conf]: Filebucketed /etc/puppet/puppet.conf.d/10-main.conf to puppet with sum 73ac1d853f8c1aeb622e53a03efe0b07
Notice: /Stage[main]/Base::Puppet/Base::Puppet::Config[main]/File[/etc/puppet/puppet.conf.d/10-main.conf]/content: content changed '{md5}73ac1d853f8c1aeb622e53a03efe0b07' to '{md5}114e8c74812328453d3100c7f0443dbe'
Info: /Stage[main]/Base::Puppet/Base::Puppet::Config[main]/File[/etc/puppet/puppet.conf.d/10-main.conf]: Scheduling refresh of Exec[delete master certs]
Info: /Stage[main]/Base::Puppet/Base::Puppet::Config[main]/File[/etc/puppet/puppet.conf.d/10-main.conf]: Scheduling refresh of Exec[compile puppet.conf]
Notice: /Stage[main]/Base::Puppet/Exec[delete master certs]: Triggered 'refresh' from 1 events
Notice: /Stage[main]/Base::Puppet/Exec[compile puppet.conf]: Triggered 'refresh' from 1 events
Notice: Applied catalog in 6.15 seconds
root@deployment-alex-test:/var/lib/puppet# puppet agent -tv
Info: Creating a new SSL key for deployment-alex-test.deployment-prep.eqiad.wmflabs
Info: Caching certificate for ca
Info: csr_attributes file loading from /etc/puppet/csr_attributes.yaml
Info: Creating a new SSL certificate request for deployment-alex-test.deployment-prep.eqiad.wmflabs
Info: Certificate Request fingerprint (SHA256): 32:94:DC:A9:6A:18:DD:62:90:BF:F8:A7:BF:74:3E:11:84:E1:E9:4A:42:99:E2:26:CD:55:E9:09:F5:9D:FE:BF
Info: Caching certificate for deployment-alex-test.deployment-prep.eqiad.wmflabs
Info: Caching certificate_revocation_list for ca
Info: Caching certificate for ca
Info: Using configured environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
[nb: snip deployment-prep cherry-picks diff from paste]
Info: Loading facts
Info: Caching catalog for deployment-alex-test.deployment-prep.eqiad.wmflabs
Notice: /Stage[main]/Base::Environment/Tidy[/var/tmp/core]: Tidying 0 files
Info: Applying configuration version '1561220545'
[nb: snip deployment-prep cherry-picks diff from paste]
Notice: openstack::clientpackages::vms::mitaka::stretch: no special configuration yet
Notice: /Stage[main]/Openstack::Clientpackages::Vms::Mitaka::Stretch/Notify[openstack::clientpackages::vms::mitaka::stretch: no special configuration yet]/message: defined 'message' as 'openstack::clientpackages::vms::mitaka::stretch: no special configuration yet'
Notice: The LDAP client stack for this host is: classic/sudoldap
Notice: /Stage[main]/Profile::Ldap::Client::Labs/Notify[LDAP client stack]/message: defined 'message' as 'The LDAP client stack for this host is: classic/sudoldap'
Notice: Applied catalog in 5.65 seconds

Change 539301 had a related patch set uploaded (by Hashar; owner: Hashar):
[operations/puppet@production] contint: add puppetmaster CA cert

https://gerrit.wikimedia.org/r/539301

hashar subscribed.

For the integration project I have set profile::base::certificates::puppet_ca_content: https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/539301/1/hieradata/labs/integration/common.yaml

On a freshly created instance, I can see the certificate is applied:

2019-09-26T11:14:44.681455+00:00 host-172-16-6-197 rc.local[479]: #033[mNotice: /Stage[main]/Profile::Base::Certificates/Sslcert::Ca[Puppet_Internal_CA]/File[/usr/local/share/ca-certificates/Puppet_Internal_CA.crt]/ensure: defined content as '{md5}c7de6a5f2f774dcf3fc6b60eafdfd259'#033[0m
[  145.493033] rc.local[479]: [mNotice: /Stage[main]/Profile::Base::Certificates/Sslcert::Ca[Puppet_Internal_CA]/File[/usr/local/share/ca-certificates/Puppet_Internal_CA.crt]/ensure: defined content as '{md5}c7de6a5f2f774dcf3fc6b60eafdfd259'[0m
[  145.497138] rc.local[479]: [0;32mInfo: /Stage[main]/Profile::Base::Certificates/Sslcert::Ca[Puppet_Internal_CA]/File[/usr/local/share/ca-certificates/Puppet_Internal_CA.crt]: Scheduling refresh of Exec[update-ca-certificates][0m
2019-09-26T11:14:44.685187+00:00 host-172-16-6-197 puppet-agent[5975]: (/Stage[main]/Profile::Base::Certificates/Sslcert::Ca[Puppet_Internal_CA]/File[/usr/local/share/ca-certificates/Puppet_Internal_CA.crt]/ensure) defined content as '{md5}c7de6a5f2f774dcf3fc6b60eafdfd259'
2019-09-26T11:14:44.685578+00:00 host-172-16-6-197 rc.local[479]: #033[0;32mInfo: /Stage[main]/Profile::Base::Certificates/Sslcert::Ca[Puppet_Internal_CA]/File[/usr/local/share/ca-certificates/Puppet_Internal_CA.crt]: Scheduling refresh of Exec[update-ca-certificates]#033[0m
2019-09-26T11:14:44.686184+00:00 host-172-16-6-197 puppet-agent[5975]: (/Stage[main]/Profile::Base::Certificates/Sslcert::Ca[Puppet_Internal_CA]/File[/usr/local/share/ca-certificates/Puppet_Internal_CA.crt]) Scheduling refresh of Exec[update-ca-certificates]

...
Applied catalog in 95.16 seconds

On the agent /etc/ssl/certs/Puppet_Internal_CA.pem looks correct with md5sum c7de6a5f2f774dcf3fc6b60eafdfd259.

There is then another puppet run on the agent which fails with:
self signed certificate in certificate chain for /CN=Puppet CA: integration-puppetmaster01.integration.eqiad.wmflabs

[  176.017335] rc.local[479]:
+ puppet agent -t
[Warning: Unable to fetch my node definition, but the agent run will continue:
[Warning: SSL_connect returned=1 errno=0 state=error: certificate verify failed: [self signed certificate in certificate chain for /CN=Puppet CA: integration-puppetmaster01.integration.eqiad.wmflabs]
[Info: Retrieving pluginfacts
[Error: /File[/var/lib/puppet/facts.d]: Failed to generate additional resources using 'eval_generate': SSL_connect returned=1 errno=0 state=error: certificate verify failed: [self signed certificate in certificate chain for /CN=Puppet CA: integration-puppetmaster01.integration.eqiad.wmflabs]
[Error: /File[/var/lib/puppet/facts.d]: Could not evaluate: Could not retrieve file metadata for puppet:///pluginfacts: SSL_connect returned=1 errno=0 state=error: certificate verify failed: [self signed certificate in certificate chain for /CN=Puppet CA: integration-puppetmaster01.integration.eqiad.wmflabs]
[Info: Retrieving plugin
[Error: /File[/var/lib/puppet/lib]: Failed to generate additional resources using 'eval_generate': SSL_connect returned=1 errno=0 state=error: certificate verify failed: [self signed certificate in certificate chain for /CN=Puppet CA: integration-puppetmaster01.integration.eqiad.wmflabs]
[Error: /File[/var/lib/puppet/lib]: Could not evaluate: Could not retrieve file metadata for puppet:///plugins: SSL_connect returned=1 errno=0 state=error: certificate verify failed: [self signed certificate in certificate chain for /CN=Puppet CA: integration-puppetmaster01.integration.eqiad.wmflabs]
[Info: Loading facts
[Error: Could not retrieve catalog from remote server: SSL_connect returned=1 errno=0 state=error: certificate verify failed: [self signed certificate in certificate chain for /CN=Puppet CA: integration-puppetmaster01.integration.eqiad.wmflabs]
[Warning: Not using cache on failed catalog
[Error: Could not retrieve catalog; skipping run
[Error: Could not send report: SSL_connect returned=1 errno=0 state=error: certificate verify failed: [self signed certificate in certificate chain for /CN=Puppet CA: integration-puppetmaster01.integration.eqiad.wmflabs]
+ rm /etc/block-ldap-key-lookup

And I fix it on the puppet agent via rm -fR /var/lib/puppet/ssl/ which then gives me the cert from https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/539301/1/hieradata/labs/integration/common.yaml

Notice: /Stage[main]/Profile::Base::Certificates/Sslcert::Ca[Puppet_Internal_CA]/File[/usr/local/share/ca-certificates/Puppet_Internal_CA.crt]/content: 
--- /usr/local/share/ca-certificates/Puppet_Internal_CA.crt	2019-09-26 11:14:44.653342724 +0000
+++ /tmp/puppet-file20190926-19430-1qd2lzs	2019-09-26 11:26:03.108434781 +0000
@@ -1,31 +1,34 @@
 -----BEGIN CERTIFICATE-----
-MIIFYDCCA0igAwIBAgIBATANBgkqhkiG9w0BAQsFADAcMRowGAYDVQQDDBFQdXBw
-ZXQgQ0E6IHB1cHBldDAeFw0xOTA0MDQxNDU0MDVaFw0yNDA0MDMxNDU0MDVaMBwx
-GjAYBgNVBAMMEVB1cHBldCBDQTogcHVwcGV0MIICIjANBgkqhkiG9w0BAQEFAAOC
-Ag8AMIICCgKCAgEA0pqa78Os6lknVmbN/WAYTuoCUCVau2pFmyKLYyempyXuUNja
-xkxs6mQkLiDNAnTQmUM+UBMwgFmMIYfu6UnM6AdWVBv+bzGqZA7ET9pAcPYtK940
-sTzqPc0h/mYcB8J9dulhZYwX91TDYVWff6YQr1ryQf+lUtGtK3T/1teH9GA1S0GB
-S9LfeDnUgUfXnSBPT8G+duOIY/JlaQjuDeRKFBN3+t9Of4xNQxzLqm74Wq8gtPL+
-PggkSrPCz7CwE9p0jmo/TIr6X57gwcfvj9+fVXDfQ9XbMOudMgOkDAX+VXPl6VQ7
-P2cMeHhajAyBhcaPdKiUC/I2CXtx9ajigrOjk8WNrjCwDAK7kN9lytXKGVUnzwbw
-RcTCe0mg+ttp/B7X+d5G+ryooV9v8TagG0cuaViCAGkKOQXuq0YSIUkA2NfUXZRS
-vDwVXWPXYALbpwtDd1E1NXQvQJVD3ErTa2L+kkKIF3/XOUqvzQZATb27QC3gT2En
-j45vzH2dqPzRvoJ2HCWOuVIfvpWsCHyscK77rPpU+TssN/nbyYIOZf2y6YYgs17M
-hHlFSAS13AfBT7JAM9ZCGZOeZEdtFly/ei+BUBjpEI4AtdpR/WaKu19XCBcuyVZF
-Abpmd1btfQgCgqrlMQO86mjtMJBw8LfTW+A2cJVwtUUDKwhQxJWiE6pGn7MCAwEA
-AaOBrDCBqTA3BglghkgBhvhCAQ0EKgwoUHVwcGV0IFJ1YnkvT3BlblNTTCBJbnRl
-cm5hbCBDZXJ0aWZpY2F0ZTAOBgNVHQ8BAf8EBAMCAQYwDwYDVR0TAQH/BAUwAwEB
-/zAdBgNVHQ4EFgQUaZ5FEUNC6pARPNbyoJJGHivr8QkwLgYDVR0jBCcwJaEgpB4w
-HDEaMBgGA1UEAwwRUHVwcGV0IENBOiBwdXBwZXSCAQEwDQYJKoZIhvcNAQELBQAD
-ggIBALnirHYnPwUzHwYUhnEi2KKyM68cyghZAhzGUWdeKtYSKVxaSIeSTR7+PI3G
-SC55qaB0uXY2VQfe1mEL9MmpsE6T8E4BX3PHVqmRi6F6l3eUxMEH8/a8M9Y5GaIr
-XN2WeqMCsxgslDSF6EJTxXeSU8foW66GuMfc8AfFZ4H6O/+Wje/xpcWv4karxNmF
-osdgoIljFu+PrA5ZBkWvs9BDT5Zqy9FQ4fdeG1ZvoEzvMr3zV9mNgJm811DDqCiq
-vrCLo3qD6uRUm7bpmaBoG6/WbZQLQyJiNjhFIUj114gDG3ZNgDADtrgRhiNNAyJo
-e8uix7EO7HwCVRJdiPBaS0PukUUCzNyBcTUz3M7EHChawFr86lB/6itABvMeod6Z
-gRATdDJKh8b0I2ifNhlLunkhQbAPZ2aDaLuWR/wiQoRABqJbBbwZwUHCs2VZviEW
-xRoV2ynNja8h8wu/OCpI8gx+3PDXYwm5oqfo0YTu34T2YEanryW/SUkfHGQgZtQV
-5d+ERqeM2ASWaoMRyebQig4Z9tp6E7QPGHbihQpRnyih0MEX2vs7jzIzbE0MvMRM
-FtP39mTd9r/YUVx0DrgFr6yjYHV8vdxmpN6o3DdH1vnuieGmFew8PsVBirhNYmzs
-VkfnMlAuJeIoWV9c7/toNtljfvgtQmoukTTbjO5XPmaBXDT1
+MIIF6DCCA9CgAwIBAgIBATANBgkqhkiG9w0BAQsFADBKMUgwRgYDVQQDDD9QdXBw
+ZXQgQ0E6IGludGVncmF0aW9uLXB1cHBldG1hc3RlcjAxLmludGVncmF0aW9uLmVx
+aWFkLndtZmxhYnMwHhcNMTYwOTE4MjA0MDE0WhcNMjEwOTE4MjA0MDE0WjBKMUgw
+RgYDVQQDDD9QdXBwZXQgQ0E6IGludGVncmF0aW9uLXB1cHBldG1hc3RlcjAxLmlu
+dGVncmF0aW9uLmVxaWFkLndtZmxhYnMwggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAw
+ggIKAoICAQC1r2UtKYPShtb+3Oi/GVg0fVExu9D5nXDuDnjAJPtYwtOWYtCE++pJ
+g2aX7XiUBOGLqMAu5N1y3cn8m9ZxP/nr2B/o0bj6+hg3Lyp/JKI2k3UGpptJVfyQ
+VA6UQoh/t+Z+mEAefBY3xhPszP6xscny2uyaSoJ11BgFUHLM7vJpnz/Lt2BP5cIc
++KxaWgzfR//bEz9tbzDs3Kb26k2prW6cx6uuRR7iHQ347bO9AbV5YmiJM+mCLQ3W
+lpIHi6/84w8QoHSTxveFj2Hw+cSyBDiCHfTYOpzqPN7JnUqLAT4BVxsIOdbcImHc
+5hdZNlWj0a5iWVPfqVZqlHrBPoQjFTf9wnZ2B0jzgyBfq1v8lYcn1oNcMDbe8T1s
+xuzGU1eRqx/6H6VcOv0K/9goEju4TwD53F6laPNI/re7OyS2v+8SGFFGoFQBfl7z
+YTDY807KN2ptBB+n0MTRzshJbqmzOrkxlx18UD/4bLyvUnLbNm9dSX1B6pAHELyv
+rWbpynPKL0hyGimkBU0IjzWRL3GZk0nWwK4st3SvrkaYTj93WHhkuh1UlRS3K620
+/Of75OY5FTk1+JEvh6+g/TLRMvs+Dfdb/dsjk1VIXWw7Jdtc93G+ohAuclhlxTIO
+NOcsGlqStkkAVArCMcRx5Txyuj+Dp+uGNhOQGjlBxCqjuNJoC1GaawIDAQABo4HY
+MIHVMDUGCWCGSAGG+EIBDQQoUHVwcGV0IFJ1YnkvT3BlblNTTCBJbnRlcm5hbCBD
+ZXJ0aWZpY2F0ZTAOBgNVHQ8BAf8EBAMCAQYwDwYDVR0TAQH/BAUwAwEB/zAdBgNV
+HQ4EFgQU9d4k/yretmHLPsAXs9HU4L43krIwXAYDVR0jBFUwU6FOpEwwSjFIMEYG
+A1UEAww/UHVwcGV0IENBOiBpbnRlZ3JhdGlvbi1wdXBwZXRtYXN0ZXIwMS5pbnRl
+Z3JhdGlvbi5lcWlhZC53bWZsYWJzggEBMA0GCSqGSIb3DQEBCwUAA4ICAQBO4sP0
+Dyck+6tXYVSBt0ghZ8KG2uM2nq5n+uGZxgOu0k+cvktqU6Wff+y579biYF0S+bgx
+zcSytWRqUHDfv/ox+H9Gwlr7R97DUOGBorY+wP5VMKyycEuNEQMDxeAM2dcCAQrt
+dG4MeRDaUid4ABtJi0nOkrP8LFYKfk4rdoATm7X4HHosKUI3mdPYo1LqUlYrkv9g
+XffrP01itPLH+dM7jhOjxspMQmcakfL0cCVpYJd+l1exxb0pAuI/zzhxqdELGP4G
+2DFJON3UT8XEqs9jIXvowlTBZAy6+drYZAfGzH8qIohQFYCAM7DslEzIO1BIQkmZ
+YDq3zkXta5eSVLfLHpkavHtvihtpQymrLkWPVPux6FrJCsjQ+Pr/RX3VdbB1u6uh
+LVwiJyg4wytfmYj3t18TwUHFRSYMCitCrennZehrMgkqYTH2dIzlZc0cnqJnjLcW
+0pKdL6NNkjM0gEL7tcyoAItYJhdjr0EpavHIVr09prAUjihqyaKJDBtEaHnsFHPf
+VIoGV0J2ZhwX881l8KRjL8xTyOP36jspmAobDK45wynubxegDpEJAYniyJNt0jjd
+WOqMw3HEbpjwy15FHgdbe7Eoo9Q0tX3DWo8dysqPqhQ/rOskxDNTCPW4rFmWADl0
+DS+lpW7rmHnTihypNFoR1eGJIEi9jC0OrqiTHw==
 -----END CERTIFICATE-----

Info: Computing checksum on file /usr/local/share/ca-certificates/Puppet_Internal_CA.crt
Info: /Stage[main]/Profile::Base::Certificates/Sslcert::Ca[Puppet_Internal_CA]/File[/usr/local/share/ca-certificates/Puppet_Internal_CA.crt]: Filebucketed /usr/local/share/ca-certificates/Puppet_Internal_CA.crt to puppet with sum c7de6a5f2f774dcf3fc6b60eafdfd259
Notice: /Stage[main]/Profile::Base::Certificates/Sslcert::Ca[Puppet_Internal_CA]/File[/usr/local/share/ca-certificates/Puppet_Internal_CA.crt]/content: content changed '{md5}c7de6a5f2f774dcf3fc6b60eafdfd259' to '{md5}37b5dc41642cf4c2b15f09df440133fe'
Info: /Stage[main]/Profile::Base::Certificates/Sslcert::Ca[Puppet_Internal_CA]/File[/usr/local/share/ca-certificates/Puppet_Internal_CA.crt]: Scheduling refresh of Exec[update-ca-certificates]
Notice: /Stage[main]/Sslcert/Exec[update-ca-certificates]: Triggered 'refresh' from 1 events
Info: Sslcert::Ca[Puppet_Internal_CA]: Scheduling refresh of Exec[clear-old-puppet-ssl]
Notice: /Stage[main]/Profile::Base::Certificates/Exec[clear-old-puppet-ssl]: Triggered 'refresh' from 1 events

Change 539301 abandoned by Hashar:
contint: add puppetmaster CA cert

Reason:
I hvae dropped the patch since it does not solve anything ;)

https://gerrit.wikimedia.org/r/539301

aborrero triaged this task as Medium priority.Nov 22 2019, 10:18 AM
jbond claimed this task.
jbond subscribed.

tentatively closing this issue as its been some time and i have not seen any further reports but please re-open if its still an issue