Noticed that puppet is not running on metricsinfra-prometheus-2 though for example is fine on metricsinfra-prometheus-3
root@metricsinfra-prometheus-2:~# run-puppet-agent Error: The CRL issued by 'CN=Puppet CA: metricsinfra-puppetmaster-1.metricsinfra.eqiad1.wikimedia.cloud' has expired, verify time is synchronized Error: The CRL issued by 'CN=Puppet CA: metricsinfra-puppetmaster-1.metricsinfra.eqiad1.wikimedia.cloud' has expired, verify time is synchronized
The fix is trivial (remove the cached crl and run puppet agent again) though it is silly we have to do this manually (or at all)
root@metricsinfra-prometheus-2:/var/lib/puppet/ssl# mv crl.pem crl.pem.old root@metricsinfra-prometheus-2:/var/lib/puppet/ssl# run-puppet-agent Info: Using environment 'production' Info: Retrieving pluginfacts Info: Retrieving plugin ...
I rotated the Puppet CA there recently; I wonder if there's a missing step from that process to refresh the CRL?
Could be yeah, I don't know if puppet agent is actually supposed to be doing its own crl management/refresh.
On a side note and thinking out loud, do we even need the crl with 5y expiration? why not 666y?