Page MenuHomePhabricator

Extend Puppet CA Expiry date
Open, MediumPublic

Description

The puppet CA expires on Jun 29 19:36:29 2020 GMT, we should extend the date before it expires. The following guide seems like a reasonable approach

https://blog.flyingcircus.io/2017/09/01/how-to-renew-puppet-ca-and-server-certificates-in-place/

Details

Event Timeline

jbond created this task.Oct 23 2019, 2:18 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 23 2019, 2:18 PM
jbond triaged this task as Medium priority.Oct 23 2019, 2:18 PM
jbond added projects: Operations, Puppet.

Change 545573 had a related patch set uploaded (by Jbond; owner: John Bond):
[operations/puppet@production] puppet: manage localcacert in puppet

https://gerrit.wikimedia.org/r/545573

herron added a subscriber: herron.Oct 28 2019, 2:31 PM

Change 545573 merged by Jbond:
[operations/puppet@production] puppet: manage localcacert in puppet

https://gerrit.wikimedia.org/r/545573

jbond moved this task from Unsorted 💣 to Active 🚁 on the User-jbond board.Oct 30 2019, 4:53 PM

Change 548241 had a related patch set uploaded (by Jbond; owner: John Bond):
[operations/puppet@production] puppet_ca: update puppet ca with a new certificate valid for 10 years

https://gerrit.wikimedia.org/r/548241

jbond added a comment.Nov 4 2019, 1:06 PM

We also need to consider /usr/local/share/ca-certificates/Puppet_Internal_CA.crt which is linked to /etc/ssl/certs/Puppet_Internal_CA.pem. This cert is used for validating other services which rely on the puppetca. Once the new certificate is in place it is likely that we will need to restart/reload any deamons that use theses certificates so they have the new CA in memory. The major pain point here is likely MySQL

Volans added a project: DBA.Nov 4 2019, 1:11 PM

One thing to take into account: we're using certificates signed by the Puppet CA in many places:

  • the puppet client certificate exposed via puppet code, see base::expose_puppet_certs
  • certificates in the private puppet repo generated via the utils/create_ecdsa_cert script and cergen

And many places uses the Puppet CA certificate to validate those.
If I understand it correctly the current procedure involves to generate a new CA certificate with the existing private key, that means that the /usr/local/share/ca-certificates/Puppet_Internal_CA.crt (symlinked in /etc/ssl/certs/Puppet_Internal_CA.pem) certificate will be replaced by a new one with the new expiration.
This means also that every service that uses this certificate to validate client's certificates, needs to reload it from disk.
The problem I see for this are:

jcrespo added a subscriber: jcrespo.Nov 4 2019, 1:15 PM

While I cannot take credit for it, I hope my MySQL TLS envangelization efforts helped pushing for that functionality. I am testing MySQL/Percona Server 8 at the moment.

jbond added a comment.Nov 4 2019, 1:20 PM

We also need to consider /usr/local/share/ca-certificates/Puppet_Internal_CA.crt which is linked to /etc/ssl/certs/Puppet_Internal_CA.pem. This cert is used for validating other services which rely on the puppetca. Once the new certificate is in place it is likely that we will need to restart/reload any deamons that use theses certificates so they have the new CA in memory. The major pain point here is likely MySQL

note to self. /usr/local/share/ca-certificates/Puppet_Internal_CA.crt is managed by puppet and would pick up the certificate change in 548241

The major pain point here is likely MySQL

I am relatively sure that I didn't enabled strict cert checking because I knew this day would arrive (requires TLS, but allows certain invalid properties, like unmatching certs). That may simplify some of the steps. Even if that was the case, we can downgrade the replication temporarely to be in plain text withing a datacenter in a hot way, while only restarting an intermediate master for cross-dc replication- which is a hard requirement- that is how we deployed it last time.

IMPORTANT: The puppet CA cert (and correspondingly key), is used as a "master" (a failsafe in case the actual host key is not around) key for bacula backups. That is, if we lose it we won't be able to restore backups for hosts that no longer are around. This is documented under https://wikitech.wikimedia.org/wiki/Bacula#Restore_from_a_non-existent_host_(missing_private_key).

jbond added a comment.Nov 4 2019, 1:41 PM

IMPORTANT: The puppet CA cert (and correspondingly key), is used as a "master" (a failsafe in case the actual host key is not around) key for bacula backups. That is, if we lose it we won't be able to restore backups for hosts that no longer are around. This is documented under https://wikitech.wikimedia.org/wiki/Bacula#Restore_from_a_non-existent_host_(missing_private_key).

@akosiaris I only intend to update the public key with this change as such everything should still work post-change, or am i missing something else?

IMPORTANT: The puppet CA cert (and correspondingly key), is used as a "master" (a failsafe in case the actual host key is not around) key for bacula backups. That is, if we lose it we won't be able to restore backups for hosts that no longer are around. This is documented under https://wikitech.wikimedia.org/wiki/Bacula#Restore_from_a_non-existent_host_(missing_private_key).

@akosiaris I only intend to update the public key with this change as such everything should still work post-change, or am i missing something else?

That would work quite fine indeed. Thanks for clarifying it!

CDanis added a subscriber: CDanis.Nov 4 2019, 2:23 PM

In terms of identifying services that use keys issued by the puppet CA -- is it wrong to think that the following would be a complete list?

  • keys created using cergen
  • users of base::expose_puppet_certs
  • the few users we have that are referencing either puppet_ssldir() or manually hardcoding the /var/lib/puppet/ssl directory

I'd think it should be relatively straightforward to find all such users just by querying puppetdb.

jbond added a comment.Nov 4 2019, 2:42 PM

In terms of identifying services that use keys issued by the puppet CA -- is it wrong to think that the following would be a complete list?

  • keys created using cergen
  • users of base::expose_puppet_certs
  • the few users we have that are referencing either puppet_ssldir() or manually hardcoding the /var/lib/puppet/ssl directory

I'd think it should be relatively straightforward to find all such users just by querying puppetdb.

Thanks i have created a separate subtask to track this https://phabricator.wikimedia.org/T237259

Volans added a comment.Nov 4 2019, 3:14 PM

@CDanis the problem is that all of those identify clients, while for the CA validation we're mostly interested in the server side. So while that surely would help, it's a 1:1 mapping. Also there might be places that have hardcoded the path to the CA cert for validation, either in the puppet repo or, potentially, in other repos too (as a default for example, dunno).
I don't know if this CA is also used in the k8s world for example.

@jbond another thing to check is to see if we maybe ship the CA cert in our Debian build images too.

From a quick check they are not even discoverable by lsof as most likely the file is read at startup and cached in memory.

@CDanis the problem is that all of those identify clients, while for the CA validation we're mostly interested in the server side. So while that surely would help, it's a 1:1 mapping. Also there might be places that have hardcoded the path to the CA cert for validation, either in the puppet repo or, potentially, in other repos too (as a default for example, dunno).
I don't know if this CA is also used in the k8s world for example.

It is. https://gerrit.wikimedia.org/r/plugins/gitiles/operations/deployment-charts/+/master/helmfile.d/services/eqiad/sessionstore/values.yaml and https://gerrit.wikimedia.org/r/plugins/gitiles/operations/deployment-charts/+/master/helmfile.d/services/eqiad/sessionstore/values.yaml as well as the TLS support that is bound to make it to all the services using envoy.

Joe added a comment.Nov 5 2019, 6:46 AM

In terms of identifying services that use keys issued by the puppet CA -- is it wrong to think that the following would be a complete list?

  • keys created using cergen
  • users of base::expose_puppet_certs
  • the few users we have that are referencing either puppet_ssldir() or manually hardcoding the /var/lib/puppet/ssl directory

I'd think it should be relatively straightforward to find all such users just by querying puppetdb.

There are others that are just called via secret(), but if we're not recreating the CA, but rather just extending its expiration date, we don't really need to refresh any of the certificates involved, but basically the use of certificates signed by the puppet CA is all over the place.

This is one of the reasons why I would've liked to see the creation of a PKI solution before we got to apply TLS everywhere in production.

Joe added a comment.Nov 7 2019, 7:33 AM

One suggestion: shouldn't we keep the old CA cert around while transitioning?

What I mean is we should keep the current CA cert around in our approved certificates until we're sure it's not needed anymore and we've restarted/fixed any system that is using it as its CA.

jbond added a comment.EditedMon, Nov 18, 5:26 PM

considering we no have T237259 and the fact that the sooner we install the new certificate the more chance difficult services i.e. mysql servers. will naturally pick up the new certificate. Are we happy to progress with the change. If anything is missing from the plan what is missing. @Volans you gave the -1 so specificity looking for feedback from you, thanks

@jbond just to be on the safe side and to verify the theory, if possible make a quick test that the new cert in the CR is able to verify exiting puppet node certs and cergen certs.

I'm removing my -1 given that there is a shared understanding of the problem and I didn't see any opposition to the current general plan.

One suggestion: shouldn't we keep the old CA cert around while transitioning?
What I mean is we should keep the current CA cert around in our approved certificates until we're sure it's not needed anymore and we've restarted/fixed any system that is using it as its CA.

@Joe I'm not sure that it would add anything. If we ensure that existing certs are validated by the new pem what's the benefit of keeping the old one around? No software will load it explicitly as they will get the new one, it would just be part of the approved certs but any certs that is valid for the old pem should be valid for the new pem too AFAIUI. Correct me if I'm wrong.

@Volans cheers also re keeping the old cert around, for what its worth it will remain in the git history

I have tested a puppet run and made sure the debmonitor client works with the new CA. Are there any other service worth validating. specificity dose this cover of a cergen service?

puppet run

Notice i made the localcacert immutable to prevent puppet from replacing it with the old one.

idp2001 ~ % sudo disable-puppet 'testing new puppet ca'                      
idp2001 ~ % CERT=$(sudo puppet config print localcacert)
idp2001 ~ % sudo cp ${CERT}{,.bak}  
idp2001 ~ % sudo vi $CERT  # add new cert
idp2001 ~ % sudo chattr +i ${CERT}                                              
idp2001 ~ % openssl x509 -in $CERT -noout -dates -subject -issuer     
notBefore=Nov  4 12:09:38 2019 GMT
notAfter=Nov  1 12:09:38 2029 GMT
subject=CN = Puppet CA: palladium.eqiad.wmnet
issuer=CN = Puppet CA: palladium.eqiad.wmnet
idp2001 ~ % sudo enable-puppet 'testing new puppet ca'; sudo puppet agent -t ; sudo disable-puppet 'testing new puppet ca'; 
Info: Using configured environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Retrieving locales
Info: Loading facts
Info: Caching catalog for idp2001.wikimedia.org
Info: Unable to serialize catalog to json, retrying with pson
Info: Applying configuration version '(683101871c) Gehel - wdqs: fix logging configuration for updated entities'
Notice: /Stage[main]/Base::Puppet/File[/var/lib/puppet/ssl/certs/ca.pem]/content: 
--- /var/lib/puppet/ssl/certs/ca.pem    2019-11-19 10:39:20.655588263 +0000
+++ /tmp/puppet-file20191119-1406-1oebk4p 2019-11-19 10:41:02.176724020 +0000
@@ -1,31 +1,31 @@
 -----BEGIN CERTIFICATE-----
-MIIFXzCCA0egAwIBAgIUQPBPwrOR622kzKx6kBuEsU5OxV0wDQYJKoZIhvcNAQEL
-BQAwKzEpMCcGA1UEAwwgUHVwcGV0IENBOiBwYWxsYWRpdW0uZXFpYWQud21uZXQw
-HhcNMTkxMTA0MTIwOTM4WhcNMjkxMTAxMTIwOTM4WjArMSkwJwYDVQQDDCBQdXBw
-ZXQgQ0E6IHBhbGxhZGl1bS5lcWlhZC53bW5ldDCCAiIwDQYJKoZIhvcNAQEBBQAD
-ggIPADCCAgoCggIBAMMi6NogAUaQaMhR7iQfPX1hQAzHUqnIeHxaPp/JVyPxTEBy
-sjfmQsO3dkr/eRPaS+6VIljA7lc9lvbwVkUb3DUc/lmqrz0xipeWj0HvAG3Lt+Vw
-rwgQkQrtwEntIyWhVC3sVrBaPMqz42ybh+QIWWdDBR1H3LpURi1Aidd+P6zCjPS/
-9/wiujXNwrqe006JCv3M0kZqkz/0YqBxMQIn1mzw+xaZJ0pn7LJL3nq2iidS+zgO
-zuXU/Sf7yZVG6xKn/AXBmgu1NEkYJyyBjyHIXI71AW49jOSTds9NZ6kBnJOboTZC
-52Wg/1QjREjcDdXsApXDXCEtQZRCIEMK3BvRgeEcARargOgcqcPKy0QYk5Ch2BkZ
-PkKDI6DO/7mmkFA0Xs7D7KVU+wyzdMxHya+l4vEblE66imSLhA4cSJea/AmaYRBe
-SXVR0duNwvs8rq8kW3bB7lrqgd6D6pF9/OHwI3gPNDtc5Eq1tD/R1FK4VUIq1m24
-8ib0abtirnROtMuS7GCdjDrLFJb3eO+fgkpAsW3Ga7taTBJ4AqbgoxB2SXuej0bz
-Wfe2hk1tzJjY6qSlo/nbtRf+eHq75tqTK1ybcL5YUlLV2dCnUgho9porL+ms5+ay
-b4T0DGRYg+xxAUali4eLmA2PETuZILPktJCQumhP7yCAerWfNjKgZXreAsldAgMB
-AAGjezB5MA8GA1UdEwEB/wQFMAMBAf8wNwYJYIZIAYb4QgENBCoWKFB1cHBldCBS
-dWJ5L09wZW5TU0wgSW50ZXJuYWwgQ2VydGlmaWNhdGUwDgYDVR0PAQH/BAQDAgEG
-MB0GA1UdDgQWBBRZ5IYwfgKvDQCmdO2a9g4XrmyuujANBgkqhkiG9w0BAQsFAAOC
-AgEAhx2QGcCOlGIRKWmnG0zbdpOVoy1L9Bjb3EuCkGWOue1cod2BINU+65PDmMMl
-MTvoExKJI/fbs8ADGaVDAeyt2LHiOLbp8sRn6ThFmhnQN2uU61zvAwneVnCApDFO
-0+gEok/mNtD4FLKP/4OhHfcSgmw/3M3I04Nrm3ssu37jCss7ZnZ5LrVZBzT41ulc
-UZ1Y1JPSLFvdd8kA053oR3GDmchOIqWXkPBo6XjvE/dVGdoUSeWdNIAVmFvZTc1I
-/KGhkw0ll3bNIHmWRWPjRR5QmHTmJTgoxIXWZcr2vRLh3Mjyq1mLw4YEjvYPLtIR
-tBGswBpc7eY8exDDkA1tJhxKS3DA0JkGm2wbAfQU2vim54VQ09J/8wCiTsUxNT5U
-E2UwAW+fbLghjItFULr7B09usEXo6Qoiq3QGsJal1ksfjIxA8l0GY7v8l4io2Hsa
-nT6EssrHNxEEZQxY4tBp1c+qS8IG7ILyAAiwtLFRtjcp2rQRvZLDSZ7FJivrqOjY
-h4us+rUVI/KJfaKHrh70Q5ufj+dOZFBmpLgupzxP1aWNRtFHNiJqYIVcAjvba3dv
-SaEqoNHJ2+KytzdcT9HzY/ywvd0tUFBJCCtuGpwVtimHYXkInwFfP4zmFZmsETld
-Jl3aYuLUirKWSp+dQm8ikFCJ2gGaB8WHQWzIswFEw08vpAM=
+MIIFSjCCAzKgAwIBAgIBATANBgkqhkiG9w0BAQsFADArMSkwJwYDVQQDDCBQdXBw
+ZXQgQ0E6IHBhbGxhZGl1bS5lcWlhZC53bW5ldDAeFw0xNTA2MzAxOTM2MjlaFw0y
+MDA2MjkxOTM2MjlaMCsxKTAnBgNVBAMMIFB1cHBldCBDQTogcGFsbGFkaXVtLmVx
+aWFkLndtbmV0MIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEAwyLo2iAB
+RpBoyFHuJB89fWFADMdSqch4fFo+n8lXI/FMQHKyN+ZCw7d2Sv95E9pL7pUiWMDu
+Vz2W9vBWRRvcNRz+WaqvPTGKl5aPQe8Abcu35XCvCBCRCu3ASe0jJaFULexWsFo8
+yrPjbJuH5AhZZ0MFHUfculRGLUCJ134/rMKM9L/3/CK6Nc3Cup7TTokK/czSRmqT
+P/RioHExAifWbPD7FpknSmfsskveeraKJ1L7OA7O5dT9J/vJlUbrEqf8BcGaC7U0
+SRgnLIGPIchcjvUBbj2M5JN2z01nqQGck5uhNkLnZaD/VCNESNwN1ewClcNcIS1B
+lEIgQwrcG9GB4RwBFquA6Bypw8rLRBiTkKHYGRk+QoMjoM7/uaaQUDRezsPspVT7
+DLN0zEfJr6Xi8RuUTrqKZIuEDhxIl5r8CZphEF5JdVHR243C+zyuryRbdsHuWuqB
+3oPqkX384fAjeA80O1zkSrW0P9HUUrhVQirWbbjyJvRpu2KudE60y5LsYJ2MOssU
+lvd475+CSkCxbcZru1pMEngCpuCjEHZJe56PRvNZ97aGTW3MmNjqpKWj+du1F/54
+ervm2pMrXJtwvlhSUtXZ0KdSCGj2misv6azn5rJvhPQMZFiD7HEBRqWLh4uYDY8R
+O5kgs+S0kJC6aE/vIIB6tZ82MqBlet4CyV0CAwEAAaN5MHcwDwYDVR0TAQH/BAUw
+AwEB/zA1BglghkgBhvhCAQ0EKFB1cHBldCBSdWJ5L09wZW5TU0wgSW50ZXJuYWwg
+Q2VydGlmaWNhdGUwHQYDVR0OBBYEFFnkhjB+Aq8NAKZ07Zr2DheubK66MA4GA1Ud
+DwEB/wQEAwIBBjANBgkqhkiG9w0BAQsFAAOCAgEABa0riHHTnPz1e5WGpLZTjX3I
+NGRJh+SlWlbBmKxqqkNVm7chJckXErehYRLwOFFGNys4iL61wr5xYJM4jqy9fVt0
+0GzbC8gFM3qN0iHVcVp9aYSxJsmi81GIR569LrNSyyL3r9LRzRyFDd//TS8b7UMy
+CQveg9FUFFdJlJTFit/HXICudq98fMyiL36QIcBoJFmAI1kUCEVrWtFIfV91wVy0
+MwZz1x+jqEATjA7hZ9zNpIJxP6AIb4lZK0OX8GJnTNprGr8k6f5YZWN3ykUhDBCn
+DWI3lFxJhBsWdlCUGvrYFVYDO4gIpKjWLJ5eExu+u94YeCPTDprJLB1ClyWGnUrI
+wl1g2kxm9yqNEpakxzr4SsDv06IASr8jBx5f2zn/MNOk99jjF2Hx1S6wBBIDqFnK
+3VLZGNU/iQRBDvo6tfF11kQhGbYIfOt1R8dNxSeuUXBiyjVm8hZZMajJvwPPQpt6
+X8mAq+ydihHe4/T20dAxdReA2yP3IL5L1cbK5Aa0LU/ckGmHhg8lrrv/lTvDz4KZ
+n1Rzg0zOr07u9PYSzDMtFFMf0qrqLQ3KY4GJNAhg8TLAkQTpZZk+1/nZmiBG6H9q
+ar7DyLkvO1+I42W3Qsi34NISNdV1d0zery5wZ43DDhIdWO3x27mvNtqLODdKGSgZ
+Hv6CZq21bZOfTLBuO+c=
 -----END CERTIFICATE-----

Info: Computing checksum on file /var/lib/puppet/ssl/certs/ca.pem
Info: /Stage[main]/Base::Puppet/File[/var/lib/puppet/ssl/certs/ca.pem]: Filebucketed /var/lib/puppet/ssl/certs/ca.pem to puppet with sum 623323baf7381676eb054cd477139e21                                                               
Error: Operation not permitted @ rb_file_s_rename - (/var/lib/puppet/ssl/certs/ca.pem20191119-1406-ka6y6t, /var/lib/puppet/ssl/certs/ca.pem)
Notice: /Stage[main]/Base::Puppet/File[/var/lib/puppet/ssl/certs/ca.pem]/content: 
--- /var/lib/puppet/ssl/certs/ca.pem    2019-11-19 10:39:20.655588263 +0000
+++ /tmp/puppet-file20191119-1406-y0xs34  2019-11-19 10:41:02.356726032 +0000
@@ -1,31 +1,31 @@
 -----BEGIN CERTIFICATE-----
-MIIFXzCCA0egAwIBAgIUQPBPwrOR622kzKx6kBuEsU5OxV0wDQYJKoZIhvcNAQEL
-BQAwKzEpMCcGA1UEAwwgUHVwcGV0IENBOiBwYWxsYWRpdW0uZXFpYWQud21uZXQw
-HhcNMTkxMTA0MTIwOTM4WhcNMjkxMTAxMTIwOTM4WjArMSkwJwYDVQQDDCBQdXBw
-ZXQgQ0E6IHBhbGxhZGl1bS5lcWlhZC53bW5ldDCCAiIwDQYJKoZIhvcNAQEBBQAD
-ggIPADCCAgoCggIBAMMi6NogAUaQaMhR7iQfPX1hQAzHUqnIeHxaPp/JVyPxTEBy
-sjfmQsO3dkr/eRPaS+6VIljA7lc9lvbwVkUb3DUc/lmqrz0xipeWj0HvAG3Lt+Vw
-rwgQkQrtwEntIyWhVC3sVrBaPMqz42ybh+QIWWdDBR1H3LpURi1Aidd+P6zCjPS/
-9/wiujXNwrqe006JCv3M0kZqkz/0YqBxMQIn1mzw+xaZJ0pn7LJL3nq2iidS+zgO
-zuXU/Sf7yZVG6xKn/AXBmgu1NEkYJyyBjyHIXI71AW49jOSTds9NZ6kBnJOboTZC
-52Wg/1QjREjcDdXsApXDXCEtQZRCIEMK3BvRgeEcARargOgcqcPKy0QYk5Ch2BkZ
-PkKDI6DO/7mmkFA0Xs7D7KVU+wyzdMxHya+l4vEblE66imSLhA4cSJea/AmaYRBe
-SXVR0duNwvs8rq8kW3bB7lrqgd6D6pF9/OHwI3gPNDtc5Eq1tD/R1FK4VUIq1m24
-8ib0abtirnROtMuS7GCdjDrLFJb3eO+fgkpAsW3Ga7taTBJ4AqbgoxB2SXuej0bz
-Wfe2hk1tzJjY6qSlo/nbtRf+eHq75tqTK1ybcL5YUlLV2dCnUgho9porL+ms5+ay
-b4T0DGRYg+xxAUali4eLmA2PETuZILPktJCQumhP7yCAerWfNjKgZXreAsldAgMB
-AAGjezB5MA8GA1UdEwEB/wQFMAMBAf8wNwYJYIZIAYb4QgENBCoWKFB1cHBldCBS
-dWJ5L09wZW5TU0wgSW50ZXJuYWwgQ2VydGlmaWNhdGUwDgYDVR0PAQH/BAQDAgEG
-MB0GA1UdDgQWBBRZ5IYwfgKvDQCmdO2a9g4XrmyuujANBgkqhkiG9w0BAQsFAAOC
-AgEAhx2QGcCOlGIRKWmnG0zbdpOVoy1L9Bjb3EuCkGWOue1cod2BINU+65PDmMMl
-MTvoExKJI/fbs8ADGaVDAeyt2LHiOLbp8sRn6ThFmhnQN2uU61zvAwneVnCApDFO
-0+gEok/mNtD4FLKP/4OhHfcSgmw/3M3I04Nrm3ssu37jCss7ZnZ5LrVZBzT41ulc
-UZ1Y1JPSLFvdd8kA053oR3GDmchOIqWXkPBo6XjvE/dVGdoUSeWdNIAVmFvZTc1I
-/KGhkw0ll3bNIHmWRWPjRR5QmHTmJTgoxIXWZcr2vRLh3Mjyq1mLw4YEjvYPLtIR
-tBGswBpc7eY8exDDkA1tJhxKS3DA0JkGm2wbAfQU2vim54VQ09J/8wCiTsUxNT5U
-E2UwAW+fbLghjItFULr7B09usEXo6Qoiq3QGsJal1ksfjIxA8l0GY7v8l4io2Hsa
-nT6EssrHNxEEZQxY4tBp1c+qS8IG7ILyAAiwtLFRtjcp2rQRvZLDSZ7FJivrqOjY
-h4us+rUVI/KJfaKHrh70Q5ufj+dOZFBmpLgupzxP1aWNRtFHNiJqYIVcAjvba3dv
-SaEqoNHJ2+KytzdcT9HzY/ywvd0tUFBJCCtuGpwVtimHYXkInwFfP4zmFZmsETld
-Jl3aYuLUirKWSp+dQm8ikFCJ2gGaB8WHQWzIswFEw08vpAM=
+MIIFSjCCAzKgAwIBAgIBATANBgkqhkiG9w0BAQsFADArMSkwJwYDVQQDDCBQdXBw
+ZXQgQ0E6IHBhbGxhZGl1bS5lcWlhZC53bW5ldDAeFw0xNTA2MzAxOTM2MjlaFw0y
+MDA2MjkxOTM2MjlaMCsxKTAnBgNVBAMMIFB1cHBldCBDQTogcGFsbGFkaXVtLmVx
+aWFkLndtbmV0MIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEAwyLo2iAB
+RpBoyFHuJB89fWFADMdSqch4fFo+n8lXI/FMQHKyN+ZCw7d2Sv95E9pL7pUiWMDu
+Vz2W9vBWRRvcNRz+WaqvPTGKl5aPQe8Abcu35XCvCBCRCu3ASe0jJaFULexWsFo8
+yrPjbJuH5AhZZ0MFHUfculRGLUCJ134/rMKM9L/3/CK6Nc3Cup7TTokK/czSRmqT
+P/RioHExAifWbPD7FpknSmfsskveeraKJ1L7OA7O5dT9J/vJlUbrEqf8BcGaC7U0
+SRgnLIGPIchcjvUBbj2M5JN2z01nqQGck5uhNkLnZaD/VCNESNwN1ewClcNcIS1B
+lEIgQwrcG9GB4RwBFquA6Bypw8rLRBiTkKHYGRk+QoMjoM7/uaaQUDRezsPspVT7
+DLN0zEfJr6Xi8RuUTrqKZIuEDhxIl5r8CZphEF5JdVHR243C+zyuryRbdsHuWuqB
+3oPqkX384fAjeA80O1zkSrW0P9HUUrhVQirWbbjyJvRpu2KudE60y5LsYJ2MOssU
+lvd475+CSkCxbcZru1pMEngCpuCjEHZJe56PRvNZ97aGTW3MmNjqpKWj+du1F/54
+ervm2pMrXJtwvlhSUtXZ0KdSCGj2misv6azn5rJvhPQMZFiD7HEBRqWLh4uYDY8R
+O5kgs+S0kJC6aE/vIIB6tZ82MqBlet4CyV0CAwEAAaN5MHcwDwYDVR0TAQH/BAUw
+AwEB/zA1BglghkgBhvhCAQ0EKFB1cHBldCBSdWJ5L09wZW5TU0wgSW50ZXJuYWwg
+Q2VydGlmaWNhdGUwHQYDVR0OBBYEFFnkhjB+Aq8NAKZ07Zr2DheubK66MA4GA1Ud
+DwEB/wQEAwIBBjANBgkqhkiG9w0BAQsFAAOCAgEABa0riHHTnPz1e5WGpLZTjX3I
+NGRJh+SlWlbBmKxqqkNVm7chJckXErehYRLwOFFGNys4iL61wr5xYJM4jqy9fVt0
+0GzbC8gFM3qN0iHVcVp9aYSxJsmi81GIR569LrNSyyL3r9LRzRyFDd//TS8b7UMy
+CQveg9FUFFdJlJTFit/HXICudq98fMyiL36QIcBoJFmAI1kUCEVrWtFIfV91wVy0
+MwZz1x+jqEATjA7hZ9zNpIJxP6AIb4lZK0OX8GJnTNprGr8k6f5YZWN3ykUhDBCn
+DWI3lFxJhBsWdlCUGvrYFVYDO4gIpKjWLJ5eExu+u94YeCPTDprJLB1ClyWGnUrI
+wl1g2kxm9yqNEpakxzr4SsDv06IASr8jBx5f2zn/MNOk99jjF2Hx1S6wBBIDqFnK
+3VLZGNU/iQRBDvo6tfF11kQhGbYIfOt1R8dNxSeuUXBiyjVm8hZZMajJvwPPQpt6
+X8mAq+ydihHe4/T20dAxdReA2yP3IL5L1cbK5Aa0LU/ckGmHhg8lrrv/lTvDz4KZ
+n1Rzg0zOr07u9PYSzDMtFFMf0qrqLQ3KY4GJNAhg8TLAkQTpZZk+1/nZmiBG6H9q
+ar7DyLkvO1+I42W3Qsi34NISNdV1d0zery5wZ43DDhIdWO3x27mvNtqLODdKGSgZ
+Hv6CZq21bZOfTLBuO+c=
 -----END CERTIFICATE-----

Error: /Stage[main]/Base::Puppet/File[/var/lib/puppet/ssl/certs/ca.pem]/content: change from '{md5}623323baf7381676eb054cd477139e21' to '{md5}3b8edbb8e5395f7b8bfa0a5195bdf753' failed: Operation not permitted @ rb_file_s_rename - (/var/lib/puppet/ssl/certs/ca.pem20191119-1406-ka6y6t, /var/lib/puppet/ssl/certs/ca.pem)
Notice: /Stage[main]/Profile::Base::Certificates/Sslcert::Ca[Puppet_Internal_CA]/File[/usr/local/share/ca-certificates/Puppet_Internal_CA.crt]/content: 
--- /usr/local/share/ca-certificates/Puppet_Internal_CA.crt     2019-10-16 14:45:45.399318618 +0000
+++ /tmp/puppet-file20191119-1406-16gaqb4 2019-11-19 10:41:06.136768298 +0000
@@ -1,31 +1,31 @@
 -----BEGIN CERTIFICATE-----
-MIIFSjCCAzKgAwIBAgIBATANBgkqhkiG9w0BAQsFADArMSkwJwYDVQQDDCBQdXBw
-ZXQgQ0E6IHBhbGxhZGl1bS5lcWlhZC53bW5ldDAeFw0xNTA2MzAxOTM2MjlaFw0y
-MDA2MjkxOTM2MjlaMCsxKTAnBgNVBAMMIFB1cHBldCBDQTogcGFsbGFkaXVtLmVx
-aWFkLndtbmV0MIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEAwyLo2iAB
-RpBoyFHuJB89fWFADMdSqch4fFo+n8lXI/FMQHKyN+ZCw7d2Sv95E9pL7pUiWMDu
-Vz2W9vBWRRvcNRz+WaqvPTGKl5aPQe8Abcu35XCvCBCRCu3ASe0jJaFULexWsFo8
-yrPjbJuH5AhZZ0MFHUfculRGLUCJ134/rMKM9L/3/CK6Nc3Cup7TTokK/czSRmqT
-P/RioHExAifWbPD7FpknSmfsskveeraKJ1L7OA7O5dT9J/vJlUbrEqf8BcGaC7U0
-SRgnLIGPIchcjvUBbj2M5JN2z01nqQGck5uhNkLnZaD/VCNESNwN1ewClcNcIS1B
-lEIgQwrcG9GB4RwBFquA6Bypw8rLRBiTkKHYGRk+QoMjoM7/uaaQUDRezsPspVT7
-DLN0zEfJr6Xi8RuUTrqKZIuEDhxIl5r8CZphEF5JdVHR243C+zyuryRbdsHuWuqB
-3oPqkX384fAjeA80O1zkSrW0P9HUUrhVQirWbbjyJvRpu2KudE60y5LsYJ2MOssU
-lvd475+CSkCxbcZru1pMEngCpuCjEHZJe56PRvNZ97aGTW3MmNjqpKWj+du1F/54
-ervm2pMrXJtwvlhSUtXZ0KdSCGj2misv6azn5rJvhPQMZFiD7HEBRqWLh4uYDY8R
-O5kgs+S0kJC6aE/vIIB6tZ82MqBlet4CyV0CAwEAAaN5MHcwDwYDVR0TAQH/BAUw
-AwEB/zA1BglghkgBhvhCAQ0EKFB1cHBldCBSdWJ5L09wZW5TU0wgSW50ZXJuYWwg
-Q2VydGlmaWNhdGUwHQYDVR0OBBYEFFnkhjB+Aq8NAKZ07Zr2DheubK66MA4GA1Ud
-DwEB/wQEAwIBBjANBgkqhkiG9w0BAQsFAAOCAgEABa0riHHTnPz1e5WGpLZTjX3I
-NGRJh+SlWlbBmKxqqkNVm7chJckXErehYRLwOFFGNys4iL61wr5xYJM4jqy9fVt0
-0GzbC8gFM3qN0iHVcVp9aYSxJsmi81GIR569LrNSyyL3r9LRzRyFDd//TS8b7UMy
-CQveg9FUFFdJlJTFit/HXICudq98fMyiL36QIcBoJFmAI1kUCEVrWtFIfV91wVy0
-MwZz1x+jqEATjA7hZ9zNpIJxP6AIb4lZK0OX8GJnTNprGr8k6f5YZWN3ykUhDBCn
-DWI3lFxJhBsWdlCUGvrYFVYDO4gIpKjWLJ5eExu+u94YeCPTDprJLB1ClyWGnUrI
-wl1g2kxm9yqNEpakxzr4SsDv06IASr8jBx5f2zn/MNOk99jjF2Hx1S6wBBIDqFnK
-3VLZGNU/iQRBDvo6tfF11kQhGbYIfOt1R8dNxSeuUXBiyjVm8hZZMajJvwPPQpt6
-X8mAq+ydihHe4/T20dAxdReA2yP3IL5L1cbK5Aa0LU/ckGmHhg8lrrv/lTvDz4KZ
-n1Rzg0zOr07u9PYSzDMtFFMf0qrqLQ3KY4GJNAhg8TLAkQTpZZk+1/nZmiBG6H9q
-ar7DyLkvO1+I42W3Qsi34NISNdV1d0zery5wZ43DDhIdWO3x27mvNtqLODdKGSgZ
-Hv6CZq21bZOfTLBuO+c=
+MIIFXzCCA0egAwIBAgIUQPBPwrOR622kzKx6kBuEsU5OxV0wDQYJKoZIhvcNAQEL
+BQAwKzEpMCcGA1UEAwwgUHVwcGV0IENBOiBwYWxsYWRpdW0uZXFpYWQud21uZXQw
+HhcNMTkxMTA0MTIwOTM4WhcNMjkxMTAxMTIwOTM4WjArMSkwJwYDVQQDDCBQdXBw
+ZXQgQ0E6IHBhbGxhZGl1bS5lcWlhZC53bW5ldDCCAiIwDQYJKoZIhvcNAQEBBQAD
+ggIPADCCAgoCggIBAMMi6NogAUaQaMhR7iQfPX1hQAzHUqnIeHxaPp/JVyPxTEBy
+sjfmQsO3dkr/eRPaS+6VIljA7lc9lvbwVkUb3DUc/lmqrz0xipeWj0HvAG3Lt+Vw
+rwgQkQrtwEntIyWhVC3sVrBaPMqz42ybh+QIWWdDBR1H3LpURi1Aidd+P6zCjPS/
+9/wiujXNwrqe006JCv3M0kZqkz/0YqBxMQIn1mzw+xaZJ0pn7LJL3nq2iidS+zgO
+zuXU/Sf7yZVG6xKn/AXBmgu1NEkYJyyBjyHIXI71AW49jOSTds9NZ6kBnJOboTZC
+52Wg/1QjREjcDdXsApXDXCEtQZRCIEMK3BvRgeEcARargOgcqcPKy0QYk5Ch2BkZ
+PkKDI6DO/7mmkFA0Xs7D7KVU+wyzdMxHya+l4vEblE66imSLhA4cSJea/AmaYRBe
+SXVR0duNwvs8rq8kW3bB7lrqgd6D6pF9/OHwI3gPNDtc5Eq1tD/R1FK4VUIq1m24
+8ib0abtirnROtMuS7GCdjDrLFJb3eO+fgkpAsW3Ga7taTBJ4AqbgoxB2SXuej0bz
+Wfe2hk1tzJjY6qSlo/nbtRf+eHq75tqTK1ybcL5YUlLV2dCnUgho9porL+ms5+ay
+b4T0DGRYg+xxAUali4eLmA2PETuZILPktJCQumhP7yCAerWfNjKgZXreAsldAgMB
+AAGjezB5MA8GA1UdEwEB/wQFMAMBAf8wNwYJYIZIAYb4QgENBCoWKFB1cHBldCBS
+dWJ5L09wZW5TU0wgSW50ZXJuYWwgQ2VydGlmaWNhdGUwDgYDVR0PAQH/BAQDAgEG
+MB0GA1UdDgQWBBRZ5IYwfgKvDQCmdO2a9g4XrmyuujANBgkqhkiG9w0BAQsFAAOC
+AgEAhx2QGcCOlGIRKWmnG0zbdpOVoy1L9Bjb3EuCkGWOue1cod2BINU+65PDmMMl
+MTvoExKJI/fbs8ADGaVDAeyt2LHiOLbp8sRn6ThFmhnQN2uU61zvAwneVnCApDFO
+0+gEok/mNtD4FLKP/4OhHfcSgmw/3M3I04Nrm3ssu37jCss7ZnZ5LrVZBzT41ulc
+UZ1Y1JPSLFvdd8kA053oR3GDmchOIqWXkPBo6XjvE/dVGdoUSeWdNIAVmFvZTc1I
+/KGhkw0ll3bNIHmWRWPjRR5QmHTmJTgoxIXWZcr2vRLh3Mjyq1mLw4YEjvYPLtIR
+tBGswBpc7eY8exDDkA1tJhxKS3DA0JkGm2wbAfQU2vim54VQ09J/8wCiTsUxNT5U
+E2UwAW+fbLghjItFULr7B09usEXo6Qoiq3QGsJal1ksfjIxA8l0GY7v8l4io2Hsa
+nT6EssrHNxEEZQxY4tBp1c+qS8IG7ILyAAiwtLFRtjcp2rQRvZLDSZ7FJivrqOjY
+h4us+rUVI/KJfaKHrh70Q5ufj+dOZFBmpLgupzxP1aWNRtFHNiJqYIVcAjvba3dv
+SaEqoNHJ2+KytzdcT9HzY/ywvd0tUFBJCCtuGpwVtimHYXkInwFfP4zmFZmsETld
+Jl3aYuLUirKWSp+dQm8ikFCJ2gGaB8WHQWzIswFEw08vpAM=
 -----END CERTIFICATE-----

Info: Computing checksum on file /usr/local/share/ca-certificates/Puppet_Internal_CA.crt
Info: /Stage[main]/Profile::Base::Certificates/Sslcert::Ca[Puppet_Internal_CA]/File[/usr/local/share/ca-certificates/Puppet_Internal_CA.crt]: Filebucketed /usr/local/share/ca-certificates/Puppet_Internal_CA.crt to puppet with sum 3b8edbb8e5395f7b8bfa0a5195bdf753                                                                                      
Notice: /Stage[main]/Profile::Base::Certificates/Sslcert::Ca[Puppet_Internal_CA]/File[/usr/local/share/ca-certificates/Puppet_Internal_CA.crt]/content: content changed '{md5}3b8edbb8e5395f7b8bfa0a5195bdf753' to '{md5}623323baf7381676eb054cd477139e21'
Info: /Stage[main]/Profile::Base::Certificates/Sslcert::Ca[Puppet_Internal_CA]/File[/usr/local/share/ca-certificates/Puppet_Internal_CA.crt]: Scheduling refresh of Exec[update-ca-certificates]                                        
Notice: tlsproxy::localssl instance idp on port 443 with server name idp2001.wikimedia.org is the default server.
Notice: /Stage[main]/Profile::Idp/Tlsproxy::Localssl[idp]/Notify[tlsproxy localssl default_server on port 443]/message: defined 'message' as 'tlsproxy::localssl instance idp on port 443 with server name idp2001.wikimedia.org is the default server.'
Notice: /Stage[main]/Sslcert/Exec[update-ca-certificates]: Triggered 'refresh' from 1 event
Info: Stage[main]: Unscheduling all events on Stage[main]
Notice: Applied catalog in 17.52 seconds
idp2001 ~ % sudo enable-puppet 'testing new puppet ca'; sudo puppet agent -t ; sudo disable-puppet 'testing new puppet ca';
Info: Using configured environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Retrieving locales
Info: Loading facts
Info: Caching catalog for idp2001.wikimedia.org
Info: Unable to serialize catalog to json, retrying with pson
Info: Applying configuration version '(683101871c) Gehel - wdqs: fix logging configuration for updated entities'
Notice: /Stage[main]/Base::Puppet/File[/var/lib/puppet/ssl/certs/ca.pem]/content: 
--- /var/lib/puppet/ssl/certs/ca.pem    2019-11-19 10:39:20.655588263 +0000
+++ /tmp/puppet-file20191119-3019-rinors  2019-11-19 10:41:38.493129986 +0000
@@ -1,31 +1,31 @@
 -----BEGIN CERTIFICATE-----
-MIIFXzCCA0egAwIBAgIUQPBPwrOR622kzKx6kBuEsU5OxV0wDQYJKoZIhvcNAQEL
-BQAwKzEpMCcGA1UEAwwgUHVwcGV0IENBOiBwYWxsYWRpdW0uZXFpYWQud21uZXQw
-HhcNMTkxMTA0MTIwOTM4WhcNMjkxMTAxMTIwOTM4WjArMSkwJwYDVQQDDCBQdXBw
-ZXQgQ0E6IHBhbGxhZGl1bS5lcWlhZC53bW5ldDCCAiIwDQYJKoZIhvcNAQEBBQAD
-ggIPADCCAgoCggIBAMMi6NogAUaQaMhR7iQfPX1hQAzHUqnIeHxaPp/JVyPxTEBy
-sjfmQsO3dkr/eRPaS+6VIljA7lc9lvbwVkUb3DUc/lmqrz0xipeWj0HvAG3Lt+Vw
-rwgQkQrtwEntIyWhVC3sVrBaPMqz42ybh+QIWWdDBR1H3LpURi1Aidd+P6zCjPS/
-9/wiujXNwrqe006JCv3M0kZqkz/0YqBxMQIn1mzw+xaZJ0pn7LJL3nq2iidS+zgO
-zuXU/Sf7yZVG6xKn/AXBmgu1NEkYJyyBjyHIXI71AW49jOSTds9NZ6kBnJOboTZC
-52Wg/1QjREjcDdXsApXDXCEtQZRCIEMK3BvRgeEcARargOgcqcPKy0QYk5Ch2BkZ
-PkKDI6DO/7mmkFA0Xs7D7KVU+wyzdMxHya+l4vEblE66imSLhA4cSJea/AmaYRBe
-SXVR0duNwvs8rq8kW3bB7lrqgd6D6pF9/OHwI3gPNDtc5Eq1tD/R1FK4VUIq1m24
-8ib0abtirnROtMuS7GCdjDrLFJb3eO+fgkpAsW3Ga7taTBJ4AqbgoxB2SXuej0bz
-Wfe2hk1tzJjY6qSlo/nbtRf+eHq75tqTK1ybcL5YUlLV2dCnUgho9porL+ms5+ay
-b4T0DGRYg+xxAUali4eLmA2PETuZILPktJCQumhP7yCAerWfNjKgZXreAsldAgMB
-AAGjezB5MA8GA1UdEwEB/wQFMAMBAf8wNwYJYIZIAYb4QgENBCoWKFB1cHBldCBS
-dWJ5L09wZW5TU0wgSW50ZXJuYWwgQ2VydGlmaWNhdGUwDgYDVR0PAQH/BAQDAgEG
-MB0GA1UdDgQWBBRZ5IYwfgKvDQCmdO2a9g4XrmyuujANBgkqhkiG9w0BAQsFAAOC
-AgEAhx2QGcCOlGIRKWmnG0zbdpOVoy1L9Bjb3EuCkGWOue1cod2BINU+65PDmMMl
-MTvoExKJI/fbs8ADGaVDAeyt2LHiOLbp8sRn6ThFmhnQN2uU61zvAwneVnCApDFO
-0+gEok/mNtD4FLKP/4OhHfcSgmw/3M3I04Nrm3ssu37jCss7ZnZ5LrVZBzT41ulc
-UZ1Y1JPSLFvdd8kA053oR3GDmchOIqWXkPBo6XjvE/dVGdoUSeWdNIAVmFvZTc1I
-/KGhkw0ll3bNIHmWRWPjRR5QmHTmJTgoxIXWZcr2vRLh3Mjyq1mLw4YEjvYPLtIR
-tBGswBpc7eY8exDDkA1tJhxKS3DA0JkGm2wbAfQU2vim54VQ09J/8wCiTsUxNT5U
-E2UwAW+fbLghjItFULr7B09usEXo6Qoiq3QGsJal1ksfjIxA8l0GY7v8l4io2Hsa
-nT6EssrHNxEEZQxY4tBp1c+qS8IG7ILyAAiwtLFRtjcp2rQRvZLDSZ7FJivrqOjY
-h4us+rUVI/KJfaKHrh70Q5ufj+dOZFBmpLgupzxP1aWNRtFHNiJqYIVcAjvba3dv
-SaEqoNHJ2+KytzdcT9HzY/ywvd0tUFBJCCtuGpwVtimHYXkInwFfP4zmFZmsETld
-Jl3aYuLUirKWSp+dQm8ikFCJ2gGaB8WHQWzIswFEw08vpAM=
+MIIFSjCCAzKgAwIBAgIBATANBgkqhkiG9w0BAQsFADArMSkwJwYDVQQDDCBQdXBw
+ZXQgQ0E6IHBhbGxhZGl1bS5lcWlhZC53bW5ldDAeFw0xNTA2MzAxOTM2MjlaFw0y
+MDA2MjkxOTM2MjlaMCsxKTAnBgNVBAMMIFB1cHBldCBDQTogcGFsbGFkaXVtLmVx
+aWFkLndtbmV0MIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEAwyLo2iAB
+RpBoyFHuJB89fWFADMdSqch4fFo+n8lXI/FMQHKyN+ZCw7d2Sv95E9pL7pUiWMDu
+Vz2W9vBWRRvcNRz+WaqvPTGKl5aPQe8Abcu35XCvCBCRCu3ASe0jJaFULexWsFo8
+yrPjbJuH5AhZZ0MFHUfculRGLUCJ134/rMKM9L/3/CK6Nc3Cup7TTokK/czSRmqT
+P/RioHExAifWbPD7FpknSmfsskveeraKJ1L7OA7O5dT9J/vJlUbrEqf8BcGaC7U0
+SRgnLIGPIchcjvUBbj2M5JN2z01nqQGck5uhNkLnZaD/VCNESNwN1ewClcNcIS1B
+lEIgQwrcG9GB4RwBFquA6Bypw8rLRBiTkKHYGRk+QoMjoM7/uaaQUDRezsPspVT7
+DLN0zEfJr6Xi8RuUTrqKZIuEDhxIl5r8CZphEF5JdVHR243C+zyuryRbdsHuWuqB
+3oPqkX384fAjeA80O1zkSrW0P9HUUrhVQirWbbjyJvRpu2KudE60y5LsYJ2MOssU
+lvd475+CSkCxbcZru1pMEngCpuCjEHZJe56PRvNZ97aGTW3MmNjqpKWj+du1F/54
+ervm2pMrXJtwvlhSUtXZ0KdSCGj2misv6azn5rJvhPQMZFiD7HEBRqWLh4uYDY8R
+O5kgs+S0kJC6aE/vIIB6tZ82MqBlet4CyV0CAwEAAaN5MHcwDwYDVR0TAQH/BAUw
+AwEB/zA1BglghkgBhvhCAQ0EKFB1cHBldCBSdWJ5L09wZW5TU0wgSW50ZXJuYWwg
+Q2VydGlmaWNhdGUwHQYDVR0OBBYEFFnkhjB+Aq8NAKZ07Zr2DheubK66MA4GA1Ud
+DwEB/wQEAwIBBjANBgkqhkiG9w0BAQsFAAOCAgEABa0riHHTnPz1e5WGpLZTjX3I
+NGRJh+SlWlbBmKxqqkNVm7chJckXErehYRLwOFFGNys4iL61wr5xYJM4jqy9fVt0
+0GzbC8gFM3qN0iHVcVp9aYSxJsmi81GIR569LrNSyyL3r9LRzRyFDd//TS8b7UMy
+CQveg9FUFFdJlJTFit/HXICudq98fMyiL36QIcBoJFmAI1kUCEVrWtFIfV91wVy0
+MwZz1x+jqEATjA7hZ9zNpIJxP6AIb4lZK0OX8GJnTNprGr8k6f5YZWN3ykUhDBCn
+DWI3lFxJhBsWdlCUGvrYFVYDO4gIpKjWLJ5eExu+u94YeCPTDprJLB1ClyWGnUrI
+wl1g2kxm9yqNEpakxzr4SsDv06IASr8jBx5f2zn/MNOk99jjF2Hx1S6wBBIDqFnK
+3VLZGNU/iQRBDvo6tfF11kQhGbYIfOt1R8dNxSeuUXBiyjVm8hZZMajJvwPPQpt6
+X8mAq+ydihHe4/T20dAxdReA2yP3IL5L1cbK5Aa0LU/ckGmHhg8lrrv/lTvDz4KZ
+n1Rzg0zOr07u9PYSzDMtFFMf0qrqLQ3KY4GJNAhg8TLAkQTpZZk+1/nZmiBG6H9q
+ar7DyLkvO1+I42W3Qsi34NISNdV1d0zery5wZ43DDhIdWO3x27mvNtqLODdKGSgZ
+Hv6CZq21bZOfTLBuO+c=
 -----END CERTIFICATE-----

Info: Computing checksum on file /var/lib/puppet/ssl/certs/ca.pem
Info: /Stage[main]/Base::Puppet/File[/var/lib/puppet/ssl/certs/ca.pem]: Filebucketed /var/lib/puppet/ssl/certs/ca.pem to puppet with sum 623323baf7381676eb054cd477139e21                                                               
Error: Operation not permitted @ rb_file_s_rename - (/var/lib/puppet/ssl/certs/ca.pem20191119-3019-iqmsel, /var/lib/puppet/ssl/certs/ca.pem)
Notice: /Stage[main]/Base::Puppet/File[/var/lib/puppet/ssl/certs/ca.pem]/content: 
--- /var/lib/puppet/ssl/certs/ca.pem    2019-11-19 10:39:20.655588263 +0000
+++ /tmp/puppet-file20191119-3019-mv2xng  2019-11-19 10:41:38.657131819 +0000
@@ -1,31 +1,31 @@
 -----BEGIN CERTIFICATE-----
-MIIFXzCCA0egAwIBAgIUQPBPwrOR622kzKx6kBuEsU5OxV0wDQYJKoZIhvcNAQEL
-BQAwKzEpMCcGA1UEAwwgUHVwcGV0IENBOiBwYWxsYWRpdW0uZXFpYWQud21uZXQw
-HhcNMTkxMTA0MTIwOTM4WhcNMjkxMTAxMTIwOTM4WjArMSkwJwYDVQQDDCBQdXBw
-ZXQgQ0E6IHBhbGxhZGl1bS5lcWlhZC53bW5ldDCCAiIwDQYJKoZIhvcNAQEBBQAD
-ggIPADCCAgoCggIBAMMi6NogAUaQaMhR7iQfPX1hQAzHUqnIeHxaPp/JVyPxTEBy
-sjfmQsO3dkr/eRPaS+6VIljA7lc9lvbwVkUb3DUc/lmqrz0xipeWj0HvAG3Lt+Vw
-rwgQkQrtwEntIyWhVC3sVrBaPMqz42ybh+QIWWdDBR1H3LpURi1Aidd+P6zCjPS/
-9/wiujXNwrqe006JCv3M0kZqkz/0YqBxMQIn1mzw+xaZJ0pn7LJL3nq2iidS+zgO
-zuXU/Sf7yZVG6xKn/AXBmgu1NEkYJyyBjyHIXI71AW49jOSTds9NZ6kBnJOboTZC
-52Wg/1QjREjcDdXsApXDXCEtQZRCIEMK3BvRgeEcARargOgcqcPKy0QYk5Ch2BkZ
-PkKDI6DO/7mmkFA0Xs7D7KVU+wyzdMxHya+l4vEblE66imSLhA4cSJea/AmaYRBe
-SXVR0duNwvs8rq8kW3bB7lrqgd6D6pF9/OHwI3gPNDtc5Eq1tD/R1FK4VUIq1m24
-8ib0abtirnROtMuS7GCdjDrLFJb3eO+fgkpAsW3Ga7taTBJ4AqbgoxB2SXuej0bz
-Wfe2hk1tzJjY6qSlo/nbtRf+eHq75tqTK1ybcL5YUlLV2dCnUgho9porL+ms5+ay
-b4T0DGRYg+xxAUali4eLmA2PETuZILPktJCQumhP7yCAerWfNjKgZXreAsldAgMB
-AAGjezB5MA8GA1UdEwEB/wQFMAMBAf8wNwYJYIZIAYb4QgENBCoWKFB1cHBldCBS
-dWJ5L09wZW5TU0wgSW50ZXJuYWwgQ2VydGlmaWNhdGUwDgYDVR0PAQH/BAQDAgEG
-MB0GA1UdDgQWBBRZ5IYwfgKvDQCmdO2a9g4XrmyuujANBgkqhkiG9w0BAQsFAAOC
-AgEAhx2QGcCOlGIRKWmnG0zbdpOVoy1L9Bjb3EuCkGWOue1cod2BINU+65PDmMMl
-MTvoExKJI/fbs8ADGaVDAeyt2LHiOLbp8sRn6ThFmhnQN2uU61zvAwneVnCApDFO
-0+gEok/mNtD4FLKP/4OhHfcSgmw/3M3I04Nrm3ssu37jCss7ZnZ5LrVZBzT41ulc
-UZ1Y1JPSLFvdd8kA053oR3GDmchOIqWXkPBo6XjvE/dVGdoUSeWdNIAVmFvZTc1I
-/KGhkw0ll3bNIHmWRWPjRR5QmHTmJTgoxIXWZcr2vRLh3Mjyq1mLw4YEjvYPLtIR
-tBGswBpc7eY8exDDkA1tJhxKS3DA0JkGm2wbAfQU2vim54VQ09J/8wCiTsUxNT5U
-E2UwAW+fbLghjItFULr7B09usEXo6Qoiq3QGsJal1ksfjIxA8l0GY7v8l4io2Hsa
-nT6EssrHNxEEZQxY4tBp1c+qS8IG7ILyAAiwtLFRtjcp2rQRvZLDSZ7FJivrqOjY
-h4us+rUVI/KJfaKHrh70Q5ufj+dOZFBmpLgupzxP1aWNRtFHNiJqYIVcAjvba3dv
-SaEqoNHJ2+KytzdcT9HzY/ywvd0tUFBJCCtuGpwVtimHYXkInwFfP4zmFZmsETld
-Jl3aYuLUirKWSp+dQm8ikFCJ2gGaB8WHQWzIswFEw08vpAM=
+MIIFSjCCAzKgAwIBAgIBATANBgkqhkiG9w0BAQsFADArMSkwJwYDVQQDDCBQdXBw
+ZXQgQ0E6IHBhbGxhZGl1bS5lcWlhZC53bW5ldDAeFw0xNTA2MzAxOTM2MjlaFw0y
+MDA2MjkxOTM2MjlaMCsxKTAnBgNVBAMMIFB1cHBldCBDQTogcGFsbGFkaXVtLmVx
+aWFkLndtbmV0MIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEAwyLo2iAB
+RpBoyFHuJB89fWFADMdSqch4fFo+n8lXI/FMQHKyN+ZCw7d2Sv95E9pL7pUiWMDu
+Vz2W9vBWRRvcNRz+WaqvPTGKl5aPQe8Abcu35XCvCBCRCu3ASe0jJaFULexWsFo8
+yrPjbJuH5AhZZ0MFHUfculRGLUCJ134/rMKM9L/3/CK6Nc3Cup7TTokK/czSRmqT
+P/RioHExAifWbPD7FpknSmfsskveeraKJ1L7OA7O5dT9J/vJlUbrEqf8BcGaC7U0
+SRgnLIGPIchcjvUBbj2M5JN2z01nqQGck5uhNkLnZaD/VCNESNwN1ewClcNcIS1B
+lEIgQwrcG9GB4RwBFquA6Bypw8rLRBiTkKHYGRk+QoMjoM7/uaaQUDRezsPspVT7
+DLN0zEfJr6Xi8RuUTrqKZIuEDhxIl5r8CZphEF5JdVHR243C+zyuryRbdsHuWuqB
+3oPqkX384fAjeA80O1zkSrW0P9HUUrhVQirWbbjyJvRpu2KudE60y5LsYJ2MOssU
+lvd475+CSkCxbcZru1pMEngCpuCjEHZJe56PRvNZ97aGTW3MmNjqpKWj+du1F/54
+ervm2pMrXJtwvlhSUtXZ0KdSCGj2misv6azn5rJvhPQMZFiD7HEBRqWLh4uYDY8R
+O5kgs+S0kJC6aE/vIIB6tZ82MqBlet4CyV0CAwEAAaN5MHcwDwYDVR0TAQH/BAUw
+AwEB/zA1BglghkgBhvhCAQ0EKFB1cHBldCBSdWJ5L09wZW5TU0wgSW50ZXJuYWwg
+Q2VydGlmaWNhdGUwHQYDVR0OBBYEFFnkhjB+Aq8NAKZ07Zr2DheubK66MA4GA1Ud
+DwEB/wQEAwIBBjANBgkqhkiG9w0BAQsFAAOCAgEABa0riHHTnPz1e5WGpLZTjX3I
+NGRJh+SlWlbBmKxqqkNVm7chJckXErehYRLwOFFGNys4iL61wr5xYJM4jqy9fVt0
+0GzbC8gFM3qN0iHVcVp9aYSxJsmi81GIR569LrNSyyL3r9LRzRyFDd//TS8b7UMy
+CQveg9FUFFdJlJTFit/HXICudq98fMyiL36QIcBoJFmAI1kUCEVrWtFIfV91wVy0
+MwZz1x+jqEATjA7hZ9zNpIJxP6AIb4lZK0OX8GJnTNprGr8k6f5YZWN3ykUhDBCn
+DWI3lFxJhBsWdlCUGvrYFVYDO4gIpKjWLJ5eExu+u94YeCPTDprJLB1ClyWGnUrI
+wl1g2kxm9yqNEpakxzr4SsDv06IASr8jBx5f2zn/MNOk99jjF2Hx1S6wBBIDqFnK
+3VLZGNU/iQRBDvo6tfF11kQhGbYIfOt1R8dNxSeuUXBiyjVm8hZZMajJvwPPQpt6
+X8mAq+ydihHe4/T20dAxdReA2yP3IL5L1cbK5Aa0LU/ckGmHhg8lrrv/lTvDz4KZ
+n1Rzg0zOr07u9PYSzDMtFFMf0qrqLQ3KY4GJNAhg8TLAkQTpZZk+1/nZmiBG6H9q
+ar7DyLkvO1+I42W3Qsi34NISNdV1d0zery5wZ43DDhIdWO3x27mvNtqLODdKGSgZ
+Hv6CZq21bZOfTLBuO+c=
 -----END CERTIFICATE-----

Error: /Stage[main]/Base::Puppet/File[/var/lib/puppet/ssl/certs/ca.pem]/content: change from '{md5}623323baf7381676eb054cd477139e21' to '{md5}3b8edbb8e5395f7b8bfa0a5195bdf753' failed: Operation not permitted @ rb_file_s_rename - (/var/lib/puppet/ssl/certs/ca.pem20191119-3019-iqmsel, /var/lib/puppet/ssl/certs/ca.pem)
Notice: tlsproxy::localssl instance idp on port 443 with server name idp2001.wikimedia.org is the default server.
Notice: /Stage[main]/Profile::Idp/Tlsproxy::Localssl[idp]/Notify[tlsproxy localssl default_server on port 443]/message: defined 'message' as 'tlsproxy::localssl instance idp on port 443 with server name idp2001.wikimedia.org is the default server.'
Info: Stage[main]: Unscheduling all events on Stage[main]
Notice: Applied catalog in 15.99 seconds

debmonitor

idp2001 ~ % openssl x509 -in /etc/ssl/certs/Puppet_Internal_CA.pem -noout -dates -issuer -subject      
notBefore=Nov  4 12:09:38 2019 GMT
notAfter=Nov  1 12:09:38 2029 GMT
issuer=CN = Puppet CA: palladium.eqiad.wmnet
subject=CN = Puppet CA: palladium.eqiad.wmnet
idp2001 ~ % sudo /usr/bin/debmonitor-client                                  
INFO:debmonitor:Found 571 installed binary packages
INFO:debmonitor:Found 18 upgradable binary packages (including new dependencies)
INFO:debmonitor:Successfully sent the full update to the DebMonitor server

from the debmopnitor access logs

208.80.153.23 - - [19/Nov/2019:10:53:24 +0000] "POST /hosts/idp2001.wikimedia.org/update HTTP/1.1" 201 0 "-" "python-requests/2.21.0"

The debmonitor test didn't test much as the debmonitor client sends the puppet client cert (not the CA) and it's the server that validates it with the CA.

For cergen I would just try to validate any certificate in the private puppet repo with the new CA pem I guess.

jbond added subscribers: elukey, Ottomata.EditedTue, Nov 19, 2:35 PM

Thanks Ricardo, I have now validated all the certificates in the private repo

puppetmaster1001 ~ % openssl x509 -in new_ca.pem -noout -subject -dates -issuer 
subject=CN = Puppet CA: palladium.eqiad.wmnet
notBefore=Nov  4 12:09:38 2019 GMT
notAfter=Nov  1 12:09:38 2029 GMT
issuer=CN = Puppet CA: palladium.eqiad.wmnet
puppetmaster1001 ~ % find /srv/private/modules/secret/secrets/certificates -not -name ca.crt.pem -not -path "*/hadoop_analytics-*hadoop/*"   -name \*.crt.pem  -exec openssl verify -CAfile new_ca.pem {} \;
/srv/private/modules/secret/secrets/certificates/planet.discovery.wmnet/planet.discovery.wmnet.crt.pem: OK
/srv/private/modules/secret/secrets/certificates/swift_eqiad/swift_eqiad.crt.pem: OK
/srv/private/modules/secret/secrets/certificates/kafka_main-eqiad_broker/kafka_main-eqiad_broker.crt.pem: OK
/srv/private/modules/secret/secrets/certificates/performance.discovery.wmnet/performance.discovery.wmnet.crt.pem: OK
/srv/private/modules/secret/secrets/certificates/kafka_logging-eqiad_broker/kafka_logging-eqiad_broker.crt.pem: OK
/srv/private/modules/secret/secrets/certificates/releases.discovery.wmnet/releases.discovery.wmnet.crt.pem: OK
/srv/private/modules/secret/secrets/certificates/rt.discovery.wmnet/rt.discovery.wmnet.crt.pem: OK
/srv/private/modules/secret/secrets/certificates/graphite.discovery.wmnet/graphite.discovery.wmnet.crt.pem: OK
/srv/private/modules/secret/secrets/certificates/kafka_jumbo-eqiad_broker/kafka_jumbo-eqiad_broker.crt.pem: OK
/srv/private/modules/secret/secrets/certificates/parsoid.svc.eqiad.wmnet/parsoid.svc.eqiad.wmnet.crt.pem: OK
/srv/private/modules/secret/secrets/certificates/kafka_mirror_maker/kafka_mirror_maker.crt.pem: OK
/srv/private/modules/secret/secrets/certificates/phabricator.discovery.wmnet/phabricator.discovery.wmnet.crt.pem: OK
/srv/private/modules/secret/secrets/certificates/webserver-misc-static.discovery.wmnet/webserver-misc-static.discovery.wmnet.crt.pem: OK
/srv/private/modules/secret/secrets/certificates/grafana.discovery.wmnet/grafana.discovery.wmnet.crt.pem: OK
/srv/private/modules/secret/secrets/certificates/mwmaint.discovery.wmnet/mwmaint.discovery.wmnet.crt.pem: OK
/srv/private/modules/secret/secrets/certificates/kibana.discovery.wmnet/kibana.discovery.wmnet.crt.pem: OK
/srv/private/modules/secret/secrets/certificates/blubberoid.discovery.wmnet/blubberoid.discovery.wmnet.crt.pem: OK
/srv/private/modules/secret/secrets/certificates/puppetboard.discovery.wmnet/puppetboard.discovery.wmnet.crt.pem: OK
/srv/private/modules/secret/secrets/certificates/yarn.wikimedia.org/yarn.wikimedia.org.crt.pem: OK
/srv/private/modules/secret/secrets/certificates/etherpad.discovery.wmnet/etherpad.discovery.wmnet.crt.pem: OK
/srv/private/modules/secret/secrets/certificates/echostore.discovery.wmnet/echostore.discovery.wmnet.crt.pem: OK
/srv/private/modules/secret/secrets/certificates/peopleweb.discovery.wmnet/peopleweb.discovery.wmnet.crt.pem: OK
/srv/private/modules/secret/secrets/certificates/docker-registry.discovery.wmnet/docker-registry.discovery.wmnet.crt.pem: OK
/srv/private/modules/secret/secrets/certificates/labweb.discovery.wmnet/labweb.discovery.wmnet.crt.pem: OK
/srv/private/modules/secret/secrets/certificates/webserver-misc-apps.discovery.wmnet/webserver-misc-apps.discovery.wmnet.crt.pem: OK
/srv/private/modules/secret/secrets/certificates/debmonitor.discovery.wmnet/debmonitor.discovery.wmnet.crt.pem: OK
/srv/private/modules/secret/secrets/certificates/swift_codfw/swift_codfw.crt.pem: OK
/srv/private/modules/secret/secrets/certificates/kafka_fundraising_client/kafka_fundraising_client.crt.pem: OK
/srv/private/modules/secret/secrets/certificates/wdqs.discovery.wmnet/wdqs.discovery.wmnet.crt.pem: OK
/srv/private/modules/secret/secrets/certificates/parsoid.svc.codfw.wmnet/parsoid.svc.codfw.wmnet.crt.pem: OK
/srv/private/modules/secret/secrets/certificates/restbase.discovery.wmnet/restbase.discovery.wmnet.crt.pem: OK
/srv/private/modules/secret/secrets/certificates/kafka_logging-codfw_broker/kafka_logging-codfw_broker.crt.pem: OK
/srv/private/modules/secret/secrets/certificates/eventgate-logging-external.discovery.wmnet/eventgate-logging-external.discovery.wmnet.crt.pem: OK
/srv/private/modules/secret/secrets/certificates/kafka_main-codfw_broker/kafka_main-codfw_broker.crt.pem: OK
/srv/private/modules/secret/secrets/certificates/varnishkafka/varnishkafka.crt.pem: OK
/srv/private/modules/secret/secrets/certificates/webperf.discovery.wmnet/webperf.discovery.wmnet.crt.pem: OK
/srv/private/modules/secret/secrets/certificates/parsoid.discovery.wmnet/parsoid.discovery.wmnet.crt.pem: OK
/srv/private/modules/secret/secrets/certificates/kartotherian.discovery.wmnet/kartotherian.discovery.wmnet.crt.pem: OK

In doing that i noticed that all of the folders under /srv/private/modules/secret/secrets/certificate also have a copy of the current puppet ca named ca.crt.pem. As far as i can tell theses files are are not used by the by puppet and so i wonder if they can just be deleted or do i need to update them with the new certificate. Anyone shed light one this.

puppetmaster1001 ~ % find /srv/private/modules/secret/secrets/certificates -name ca.crt.pem -print  | while read cert ; do ca_path=${cert/\/srv\/private\/modules\/secret\/secrets\/certificates\/}; hash=$(openssl x509 -in $cert -noout -hash) ; subject=$(openssl x509 -in $cert -noout -subject); dates=$(openssl x509 -in $cert -noout -enddate); printf "%s\t%s\t%s\t%s\n" $subject $hash $dates $ca_path ; done | sort 
subject=CN = Puppet CA: palladium.eqiad.wmnet   c5aaad6f        notAfter=Jun 29 19:36:29 2020 GMT       blubberoid.discovery.wmnet/ca.crt.pem
subject=CN = Puppet CA: palladium.eqiad.wmnet   c5aaad6f        notAfter=Jun 29 19:36:29 2020 GMT       debmonitor.discovery.wmnet/ca.crt.pem
subject=CN = Puppet CA: palladium.eqiad.wmnet   c5aaad6f        notAfter=Jun 29 19:36:29 2020 GMT       docker-registry.discovery.wmnet/ca.crt.pem
subject=CN = Puppet CA: palladium.eqiad.wmnet   c5aaad6f        notAfter=Jun 29 19:36:29 2020 GMT       echostore.discovery.wmnet/ca.crt.pem
subject=CN = Puppet CA: palladium.eqiad.wmnet   c5aaad6f        notAfter=Jun 29 19:36:29 2020 GMT       etherpad.discovery.wmnet/ca.crt.pem
subject=CN = Puppet CA: palladium.eqiad.wmnet   c5aaad6f        notAfter=Jun 29 19:36:29 2020 GMT       eventgate-logging-external.discovery.wmnet/ca.crt.pem
subject=CN = Puppet CA: palladium.eqiad.wmnet   c5aaad6f        notAfter=Jun 29 19:36:29 2020 GMT       grafana.discovery.wmnet/ca.crt.pem
subject=CN = Puppet CA: palladium.eqiad.wmnet   c5aaad6f        notAfter=Jun 29 19:36:29 2020 GMT       graphite.discovery.wmnet/ca.crt.pem
subject=CN = Puppet CA: palladium.eqiad.wmnet   c5aaad6f        notAfter=Jun 29 19:36:29 2020 GMT       kafka_fundraising_client/ca.crt.pem
subject=CN = Puppet CA: palladium.eqiad.wmnet   c5aaad6f        notAfter=Jun 29 19:36:29 2020 GMT       kafka_jumbo-eqiad_broker/ca.crt.pem
subject=CN = Puppet CA: palladium.eqiad.wmnet   c5aaad6f        notAfter=Jun 29 19:36:29 2020 GMT       kafka_logging-codfw_broker/ca.crt.pem
subject=CN = Puppet CA: palladium.eqiad.wmnet   c5aaad6f        notAfter=Jun 29 19:36:29 2020 GMT       kafka_logging-eqiad_broker/ca.crt.pem
subject=CN = Puppet CA: palladium.eqiad.wmnet   c5aaad6f        notAfter=Jun 29 19:36:29 2020 GMT       kafka_main-codfw_broker/ca.crt.pem
subject=CN = Puppet CA: palladium.eqiad.wmnet   c5aaad6f        notAfter=Jun 29 19:36:29 2020 GMT       kafka_main-eqiad_broker/ca.crt.pem
subject=CN = Puppet CA: palladium.eqiad.wmnet   c5aaad6f        notAfter=Jun 29 19:36:29 2020 GMT       kafka_mirror_maker/ca.crt.pem
subject=CN = Puppet CA: palladium.eqiad.wmnet   c5aaad6f        notAfter=Jun 29 19:36:29 2020 GMT       kartotherian.discovery.wmnet/ca.crt.pem
subject=CN = Puppet CA: palladium.eqiad.wmnet   c5aaad6f        notAfter=Jun 29 19:36:29 2020 GMT       kibana.discovery.wmnet/ca.crt.pem
subject=CN = Puppet CA: palladium.eqiad.wmnet   c5aaad6f        notAfter=Jun 29 19:36:29 2020 GMT       labweb.discovery.wmnet/ca.crt.pem
subject=CN = Puppet CA: palladium.eqiad.wmnet   c5aaad6f        notAfter=Jun 29 19:36:29 2020 GMT       mwmaint.discovery.wmnet/ca.crt.pem
subject=CN = Puppet CA: palladium.eqiad.wmnet   c5aaad6f        notAfter=Jun 29 19:36:29 2020 GMT       parsoid.discovery.wmnet/ca.crt.pem
subject=CN = Puppet CA: palladium.eqiad.wmnet   c5aaad6f        notAfter=Jun 29 19:36:29 2020 GMT       parsoid.svc.codfw.wmnet/ca.crt.pem
subject=CN = Puppet CA: palladium.eqiad.wmnet   c5aaad6f        notAfter=Jun 29 19:36:29 2020 GMT       parsoid.svc.eqiad.wmnet/ca.crt.pem
subject=CN = Puppet CA: palladium.eqiad.wmnet   c5aaad6f        notAfter=Jun 29 19:36:29 2020 GMT       peopleweb.discovery.wmnet/ca.crt.pem
subject=CN = Puppet CA: palladium.eqiad.wmnet   c5aaad6f        notAfter=Jun 29 19:36:29 2020 GMT       performance.discovery.wmnet/ca.crt.pem
subject=CN = Puppet CA: palladium.eqiad.wmnet   c5aaad6f        notAfter=Jun 29 19:36:29 2020 GMT       phabricator.discovery.wmnet/ca.crt.pem
subject=CN = Puppet CA: palladium.eqiad.wmnet   c5aaad6f        notAfter=Jun 29 19:36:29 2020 GMT       planet.discovery.wmnet/ca.crt.pem
subject=CN = Puppet CA: palladium.eqiad.wmnet   c5aaad6f        notAfter=Jun 29 19:36:29 2020 GMT       puppetboard.discovery.wmnet/ca.crt.pem
subject=CN = Puppet CA: palladium.eqiad.wmnet   c5aaad6f        notAfter=Jun 29 19:36:29 2020 GMT       releases.discovery.wmnet/ca.crt.pem
subject=CN = Puppet CA: palladium.eqiad.wmnet   c5aaad6f        notAfter=Jun 29 19:36:29 2020 GMT       restbase.discovery.wmnet/ca.crt.pem
subject=CN = Puppet CA: palladium.eqiad.wmnet   c5aaad6f        notAfter=Jun 29 19:36:29 2020 GMT       rt.discovery.wmnet/ca.crt.pem
subject=CN = Puppet CA: palladium.eqiad.wmnet   c5aaad6f        notAfter=Jun 29 19:36:29 2020 GMT       swift_codfw/ca.crt.pem
subject=CN = Puppet CA: palladium.eqiad.wmnet   c5aaad6f        notAfter=Jun 29 19:36:29 2020 GMT       swift_eqiad/ca.crt.pem
subject=CN = Puppet CA: palladium.eqiad.wmnet   c5aaad6f        notAfter=Jun 29 19:36:29 2020 GMT       varnishkafka/ca.crt.pem
subject=CN = Puppet CA: palladium.eqiad.wmnet   c5aaad6f        notAfter=Jun 29 19:36:29 2020 GMT       wdqs.discovery.wmnet/ca.crt.pem
subject=CN = Puppet CA: palladium.eqiad.wmnet   c5aaad6f        notAfter=Jun 29 19:36:29 2020 GMT       webperf.discovery.wmnet/ca.crt.pem
subject=CN = Puppet CA: palladium.eqiad.wmnet   c5aaad6f        notAfter=Jun 29 19:36:29 2020 GMT       webserver-misc-apps.discovery.wmnet/ca.crt.pem
subject=CN = Puppet CA: palladium.eqiad.wmnet   c5aaad6f        notAfter=Jun 29 19:36:29 2020 GMT       webserver-misc-static.discovery.wmnet/ca.crt.pem
subject=CN = Puppet CA: palladium.eqiad.wmnet   c5aaad6f        notAfter=Jun 29 19:36:29 2020 GMT       yarn.wikimedia.org/ca.crt.pem

I also noticed a ca certificate for some hadoop services which are also due to expire. Wonder if @elukey or @Ottomata could advice on theses

subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Mar 13 15:50:49 2020 GMT       hadoop_analytics-test-hadoop/analytics1028.eqiad.wmnet/ca.crt.pem
subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Mar 13 15:50:49 2020 GMT       hadoop_analytics-test-hadoop/analytics1029.eqiad.wmnet/ca.crt.pem
subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Mar 13 15:50:49 2020 GMT       hadoop_analytics-test-hadoop/analytics1031.eqiad.wmnet/ca.crt.pem
subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Mar 13 15:50:49 2020 GMT       hadoop_analytics-test-hadoop/analytics1032.eqiad.wmnet/ca.crt.pem
subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Mar 13 15:50:49 2020 GMT       hadoop_analytics-test-hadoop/analytics1033.eqiad.wmnet/ca.crt.pem
subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Mar 13 15:50:49 2020 GMT       hadoop_analytics-test-hadoop/analytics1034.eqiad.wmnet/ca.crt.pem
subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Mar 13 15:50:49 2020 GMT       hadoop_analytics-test-hadoop/analytics1035.eqiad.wmnet/ca.crt.pem
subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Mar 13 15:50:49 2020 GMT       hadoop_analytics-test-hadoop/analytics1036.eqiad.wmnet/ca.crt.pem
subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Mar 13 15:50:49 2020 GMT       hadoop_analytics-test-hadoop/analytics1037.eqiad.wmnet/ca.crt.pem
subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Mar 13 15:50:49 2020 GMT       hadoop_analytics-test-hadoop/analytics1038.eqiad.wmnet/ca.crt.pem
subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Mar 13 15:50:49 2020 GMT       hadoop_analytics-test-hadoop/analytics1040.eqiad.wmnet/ca.crt.pem
subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Mar 13 15:50:49 2020 GMT       hadoop_analytics-test-hadoop/analytics1041.eqiad.wmnet/ca.crt.pem
subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Mar 13 15:50:49 2020 GMT       hadoop_analytics-test-hadoop/root_ca/ca.crt.pem
subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Oct 30 09:07:25 2020 GMT       hadoop_analytics-hadoop/analytics1042.eqiad.wmnet/ca.crt.pem
subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Oct 30 09:07:25 2020 GMT       hadoop_analytics-hadoop/analytics1043.eqiad.wmnet/ca.crt.pem
subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Oct 30 09:07:25 2020 GMT       hadoop_analytics-hadoop/analytics1044.eqiad.wmnet/ca.crt.pem
subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Oct 30 09:07:25 2020 GMT       hadoop_analytics-hadoop/analytics1045.eqiad.wmnet/ca.crt.pem
subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Oct 30 09:07:25 2020 GMT       hadoop_analytics-hadoop/analytics1046.eqiad.wmnet/ca.crt.pem
subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Oct 30 09:07:25 2020 GMT       hadoop_analytics-hadoop/analytics1047.eqiad.wmnet/ca.crt.pem
subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Oct 30 09:07:25 2020 GMT       hadoop_analytics-hadoop/analytics1048.eqiad.wmnet/ca.crt.pem
subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Oct 30 09:07:25 2020 GMT       hadoop_analytics-hadoop/analytics1049.eqiad.wmnet/ca.crt.pem
subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Oct 30 09:07:25 2020 GMT       hadoop_analytics-hadoop/analytics1050.eqiad.wmnet/ca.crt.pem
subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Oct 30 09:07:25 2020 GMT       hadoop_analytics-hadoop/analytics1051.eqiad.wmnet/ca.crt.pem
subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Oct 30 09:07:25 2020 GMT       hadoop_analytics-hadoop/analytics1052.eqiad.wmnet/ca.crt.pem
subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Oct 30 09:07:25 2020 GMT       hadoop_analytics-hadoop/analytics1053.eqiad.wmnet/ca.crt.pem
subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Oct 30 09:07:25 2020 GMT       hadoop_analytics-hadoop/analytics1054.eqiad.wmnet/ca.crt.pem
subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Oct 30 09:07:25 2020 GMT       hadoop_analytics-hadoop/analytics1055.eqiad.wmnet/ca.crt.pem
subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Oct 30 09:07:25 2020 GMT       hadoop_analytics-hadoop/analytics1056.eqiad.wmnet/ca.crt.pem
subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Oct 30 09:07:25 2020 GMT       hadoop_analytics-hadoop/analytics1057.eqiad.wmnet/ca.crt.pem
subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Oct 30 09:07:25 2020 GMT       hadoop_analytics-hadoop/analytics1058.eqiad.wmnet/ca.crt.pem
subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Oct 30 09:07:25 2020 GMT       hadoop_analytics-hadoop/analytics1059.eqiad.wmnet/ca.crt.pem
subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Oct 30 09:07:25 2020 GMT       hadoop_analytics-hadoop/analytics1060.eqiad.wmnet/ca.crt.pem
subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Oct 30 09:07:25 2020 GMT       hadoop_analytics-hadoop/analytics1061.eqiad.wmnet/ca.crt.pem
subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Oct 30 09:07:25 2020 GMT       hadoop_analytics-hadoop/analytics1062.eqiad.wmnet/ca.crt.pem
subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Oct 30 09:07:25 2020 GMT       hadoop_analytics-hadoop/analytics1063.eqiad.wmnet/ca.crt.pem
subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Oct 30 09:07:25 2020 GMT       hadoop_analytics-hadoop/analytics1064.eqiad.wmnet/ca.crt.pem
subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Oct 30 09:07:25 2020 GMT       hadoop_analytics-hadoop/analytics1065.eqiad.wmnet/ca.crt.pem
subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Oct 30 09:07:25 2020 GMT       hadoop_analytics-hadoop/analytics1066.eqiad.wmnet/ca.crt.pem
subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Oct 30 09:07:25 2020 GMT       hadoop_analytics-hadoop/analytics1067.eqiad.wmnet/ca.crt.pem
subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Oct 30 09:07:25 2020 GMT       hadoop_analytics-hadoop/analytics1068.eqiad.wmnet/ca.crt.pem
subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Oct 30 09:07:25 2020 GMT       hadoop_analytics-hadoop/analytics1069.eqiad.wmnet/ca.crt.pem
subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Oct 30 09:07:25 2020 GMT       hadoop_analytics-hadoop/analytics1070.eqiad.wmnet/ca.crt.pem
subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Oct 30 09:07:25 2020 GMT       hadoop_analytics-hadoop/analytics1071.eqiad.wmnet/ca.crt.pem
subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Oct 30 09:07:25 2020 GMT       hadoop_analytics-hadoop/analytics1072.eqiad.wmnet/ca.crt.pem
subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Oct 30 09:07:25 2020 GMT       hadoop_analytics-hadoop/analytics1073.eqiad.wmnet/ca.crt.pem
subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Oct 30 09:07:25 2020 GMT       hadoop_analytics-hadoop/analytics1074.eqiad.wmnet/ca.crt.pem
subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Oct 30 09:07:25 2020 GMT       hadoop_analytics-hadoop/analytics1075.eqiad.wmnet/ca.crt.pem
subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Oct 30 09:07:25 2020 GMT       hadoop_analytics-hadoop/analytics1076.eqiad.wmnet/ca.crt.pem
subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Oct 30 09:07:25 2020 GMT       hadoop_analytics-hadoop/analytics1077.eqiad.wmnet/ca.crt.pem
subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Oct 30 09:07:25 2020 GMT       hadoop_analytics-hadoop/an-master1001.eqiad.wmnet/ca.crt.pem
subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Oct 30 09:07:25 2020 GMT       hadoop_analytics-hadoop/an-master1002.eqiad.wmnet/ca.crt.pem
subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Oct 30 09:07:25 2020 GMT       hadoop_analytics-hadoop/an-worker1078.eqiad.wmnet/ca.crt.pem
subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Oct 30 09:07:25 2020 GMT       hadoop_analytics-hadoop/an-worker1079.eqiad.wmnet/ca.crt.pem
subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Oct 30 09:07:25 2020 GMT       hadoop_analytics-hadoop/an-worker1080.eqiad.wmnet/ca.crt.pem
subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Oct 30 09:07:25 2020 GMT       hadoop_analytics-hadoop/an-worker1081.eqiad.wmnet/ca.crt.pem
subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Oct 30 09:07:25 2020 GMT       hadoop_analytics-hadoop/an-worker1082.eqiad.wmnet/ca.crt.pem
subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Oct 30 09:07:25 2020 GMT       hadoop_analytics-hadoop/an-worker1083.eqiad.wmnet/ca.crt.pem
subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Oct 30 09:07:25 2020 GMT       hadoop_analytics-hadoop/an-worker1084.eqiad.wmnet/ca.crt.pem
subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Oct 30 09:07:25 2020 GMT       hadoop_analytics-hadoop/an-worker1085.eqiad.wmnet/ca.crt.pem
subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Oct 30 09:07:25 2020 GMT       hadoop_analytics-hadoop/an-worker1086.eqiad.wmnet/ca.crt.pem
subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Oct 30 09:07:25 2020 GMT       hadoop_analytics-hadoop/an-worker1087.eqiad.wmnet/ca.crt.pem
subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Oct 30 09:07:25 2020 GMT       hadoop_analytics-hadoop/an-worker1088.eqiad.wmnet/ca.crt.pem
subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Oct 30 09:07:25 2020 GMT       hadoop_analytics-hadoop/an-worker1089.eqiad.wmnet/ca.crt.pem
subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Oct 30 09:07:25 2020 GMT       hadoop_analytics-hadoop/an-worker1090.eqiad.wmnet/ca.crt.pem
subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Oct 30 09:07:25 2020 GMT       hadoop_analytics-hadoop/an-worker1091.eqiad.wmnet/ca.crt.pem
subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Oct 30 09:07:25 2020 GMT       hadoop_analytics-hadoop/an-worker1092.eqiad.wmnet/ca.crt.pem
subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Oct 30 09:07:25 2020 GMT       hadoop_analytics-hadoop/an-worker1093.eqiad.wmnet/ca.crt.pem
subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Oct 30 09:07:25 2020 GMT       hadoop_analytics-hadoop/an-worker1094.eqiad.wmnet/ca.crt.pem
subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Oct 30 09:07:25 2020 GMT       hadoop_analytics-hadoop/an-worker1095.eqiad.wmnet/ca.crt.pem
subject=CN = root_ca, C = US, ST = CA   3c943e56        notAfter=Oct 30 09:07:25 2020 GMT       hadoop_analytics-hadoop/root_ca/ca.crt.pem

finnaly i also notice certs for mcrouter simlar to hadoop

puppetmaster1001 ~ % find /srv/private/modules/secret/secrets/mcrouter -name ca.crt.pem -print  | while read cert ; do ca_path=${cert/\/srv\/private\/modules\/secret\/secrets\/certificates\/}; hash=$(openssl x509 -in $cert -noout -hash) ; subject=$(openssl x509 -in $cert -noout -subject); dates=$(openssl x509 -in $cert -noout -enddate); printf "%s\t%s\t%s\t%s\n" $subject $hash $dates $ca_path ; done | sort    
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/deploy1001.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/deploy2001.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mcrouter_ca/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1221.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1222.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1223.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1224.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1225.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1226.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1227.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1228.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1229.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1230.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1231.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1232.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1233.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1234.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1235.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1238.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1239.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1240.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1241.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1242.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1243.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1244.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1245.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1246.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1247.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1248.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1249.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1250.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1251.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1252.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1253.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1254.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1255.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1256.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1257.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1258.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1261.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1262.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1263.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1264.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1265.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1266.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1267.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1268.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1269.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1270.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1271.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1272.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1273.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1274.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1275.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1276.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1277.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1278.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1279.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1280.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1281.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1282.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1283.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1284.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1285.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1286.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1287.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1288.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1289.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1290.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1293.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1294.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1295.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1296.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1297.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1298.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1299.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1300.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1301.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1302.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1303.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1304.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1305.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1306.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1307.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1308.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1309.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1310.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1311.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1312.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1313.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1314.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1315.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1316.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1317.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1318.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1319.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1320.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1321.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1322.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1323.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1324.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1325.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1326.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1327.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1328.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1329.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1330.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1331.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1332.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1333.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1334.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1335.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1336.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1337.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1338.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1339.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1340.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1341.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1342.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1343.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1344.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1345.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1346.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1347.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw1348.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2135.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2136.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2137.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2138.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2139.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2140.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2141.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2142.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2143.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2144.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2145.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2146.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2147.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2150.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2151.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2152.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2153.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2154.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2155.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2156.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2157.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2158.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2159.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2160.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2161.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2162.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2163.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2164.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2165.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2166.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2167.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2168.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2169.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2170.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2171.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2172.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2173.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2174.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2175.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2176.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2177.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2178.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2179.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2180.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2181.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2182.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2183.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2184.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2185.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2186.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2187.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2188.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2189.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2190.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2191.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2192.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2193.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2194.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2195.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2196.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2197.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2198.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2199.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2200.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2201.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2202.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2203.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2204.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2205.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2206.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2207.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2208.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2209.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2210.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2211.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2212.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2214.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2215.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2216.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2217.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2218.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2219.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2220.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2221.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2222.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2223.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2224.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2225.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2226.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2227.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2228.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2229.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2230.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2231.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2232.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2233.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2234.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2235.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2236.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2237.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2238.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2239.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2240.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2241.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2242.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2243.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2244.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2245.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2246.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2247.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2248.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2249.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2250.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2251.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2252.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2253.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2254.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2255.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2256.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2257.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2258.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2259.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2260.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2261.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2262.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2263.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2264.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2265.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2266.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2267.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2268.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2269.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2270.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2271.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2272.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2273.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2274.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2275.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2276.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2277.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2278.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2279.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2280.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2281.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2282.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2283.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2284.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2285.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2286.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2287.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2288.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2289.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mw2290.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mwdebug1001.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mwdebug1002.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mwdebug2001.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mwdebug2002.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mwmaint1002.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/mwmaint2001.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/scandium.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/snapshot1005.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/snapshot1006.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/snapshot1007.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/snapshot1008.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/snapshot1009.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/wtp1025.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/wtp1026.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/wtp1027.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/wtp1028.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/wtp1029.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/wtp1030.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/wtp1031.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/wtp1032.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/wtp1033.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/wtp1034.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/wtp1035.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/wtp1036.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/wtp1037.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/wtp1038.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/wtp1039.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/wtp1040.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/wtp1041.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/wtp1042.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/wtp1043.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/wtp1044.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/wtp1045.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/wtp1046.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/wtp1047.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/wtp1048.eqiad.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/wtp2001.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/wtp2002.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/wtp2003.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/wtp2004.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/wtp2005.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/wtp2006.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/wtp2007.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/wtp2008.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/wtp2009.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/wtp2010.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/wtp2011.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/wtp2012.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/wtp2013.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/wtp2014.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/wtp2015.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/wtp2016.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/wtp2017.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/wtp2018.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/wtp2019.codfw.wmnet/ca.crt.pem
subject=CN = mcrouter_ca, ST = CA, C = US       6545b899        notAfter=May 19 09:02:45 2020 GMT       /srv/private/modules/secret/secrets/mcrouter/wtp2020.codfw.wmnet/ca.crt.pem

Hm, uh oh, @elukey we might want to recreate all those certs with a really long expiry :)

i noticed that all of the folders under /srv/private/modules/secret/secrets/certificate also have a copy of the current puppet ca named ca.crt.pem. As far as i can tell theses files are are not used by the by puppet and so i wonder if they can just be deleted or do i need to update them with the new certificate.

Each to be generated certificate just generates all possibly useful formats. The puppet ca.crt.pem likely isn't used anywhere in prod since it is already deployed on all hosts by default. You should probably keep things consistent though and just replace the ca.crt.pem files with copies of the puppet ca cert.

@jbond thanks a lot for the ping, in my todo list there is the action item of adding cert expiry checks to all the hadoop nodes (haven't had the time to do it yet). End of March seems more than 4 months away from now, I think that we have time to renew no? Is there anything else that I'd need to do to avoid blocking this task?

For the mcrouter part, I know that Fabian renewed those certs the last time, adding the expiry check to puppet, so when the time comes we should be alerted promptly.

Change 552260 had a related patch set uploaded (by Jbond; owner: John Bond):
[operations/puppet@production] cergen: add Icing check to validate the expiry date on certificates

https://gerrit.wikimedia.org/r/552260

@jbond maybe it is a good idea to disable puppet on all databases before merging the change and then trying a manual run on a single host to see how it goes.
As Jaime mentions at T236277#5631255, if we need to disable TLS within a DC, we better do it in a controlled way. Restarting codfw masters should be done in order to make sure we do keep TLS on replication between eqiad and codfw. Restarting MySQL on those should be easy as don't have production impact.
Disabling TLS on eqiad slaves will take a bit longer.

I am also unsure if we'd need to end up restarting MySQL at all, but better be safe and start with puppet disabled.

jbond added a comment.Wed, Nov 27, 9:50 AM

@jbond maybe it is a good idea to disable puppet on all databases before merging the change and then trying a manual run on a single host to see how it goes.
As Jaime mentions at T236277#5631255, if we need to disable TLS within a DC, we better do it in a controlled way. Restarting codfw masters should be done in order to make sure we do keep TLS on replication between eqiad and codfw. Restarting MySQL on those should be easy as don't have production impact.
Disabling TLS on eqiad slaves will take a bit longer.
I am also unsure if we'd need to end up restarting MySQL at all, but better be safe and start with puppet disabled.

Thanks Marostegui ill add that to the CR

Change 548241 merged by Jbond:
[operations/puppet@production] puppet_ca: update puppet ca with a new certificate valid for 10 years

https://gerrit.wikimedia.org/r/548241

Checks performed:

On a standalone slave (db2071) with no action on its master:

  • Run puppet
  • Stop slave ; start slave;
  • Stop MySQL, start MySQL

Replication worked fine and SSL has been enabled at all times.

On a master (db2107), without doing anything on its slaves (or upstream master):

  • Run puppet
  • Stop slave; start slave;
  • Stop MySQL, start mysql.

On a standalone slave (db2125) with a restarted master (db2107)

  • Run puppet
  • Stop slave ; start slave;
  • Stop MySQL, start MySQL

Replication worked fine and SSL has been enable at all times during those all tests.

Nothing has showed up on mariadb logs either

I have also run tcpdump to see if I was able to see replication in plan text and I wasn't.

I believe we are good to go and enable puppet back on the DBs, and we can track the restarts needed before July at T239791

jbond added a comment.EditedWed, Dec 4, 12:01 PM

The new certificate has been distributed

% sudo cumin 'A:all' 'openssl x509 -in $(sudo puppet config print localcacert 2>/dev/null) -noout -enddate '
===== NODE GROUP =====                                                                    
(1) mw2259.codfw.wmnet                                                                    
----- OUTPUT of 'openssl x509 -in...-noout -enddate ' -----                               
ssh: connect to host mw2259.codfw.wmnet port 22: Connection timed out                     
===== NODE GROUP =====                                                                    
(5) cloudservices2002-dev.wikimedia.org,cloudservices[1003-1004].wikimedia.org,labtestpuppetmaster2001.wikimedia.org,logstash1024.eqiad.wmnet                                       
----- OUTPUT of 'openssl x509 -in...-noout -enddate ' ----
notAfter=Jun 29 19:36:29 2020 GMT 
===== NODE GROUP =====   
*** SNIP ALL OTHER HOSTS ***
----- OUTPUT of 'openssl x509 -in...-noout -enddate ' -----                               
notAfter=Nov  1 12:09:38 2029 GMT                                                         
                                                                 

PASS:  100% (1400/1401) [00:31<00:00, 54.73hosts/s]
FAIL:    0% (1/1401) [00:31<7:19:39, 18.84s/hosts]     
0.1% (1/1401) of nodes failed to execute command 'openssl x509 -in...-noout -enddate ': mw2259.codfw.wmnet

logstash1024.eqiad.wmnet is being tested and will resolve on the next puppet run
labtestpuppetmaster2001 is a spare and have the hiera parameter manage_puppet_ca_file: false
cloudservices* currently have a broken puppet

jbond added a comment.Wed, Dec 4, 12:03 PM

get the same results from the following

sudo cumin 'A:all' 'openssl x509 -in /etc/ssl/certs/Puppet_Internal_CA.pem -noout -enddate '