Puppetmaster1001 is not only a puppet master but the ca server so we need to be very cautious.

Typically recreating an agent cert is along these lines:

1. on agent: puppet agent --disable "regenerating cert"
2. on agent: mv /var/lib/puppet/ssl /var/lib/puppet/ssl.orig
3. on master: puppet cert clean fqdn_of_agent
4. on agent: puppet agent --enable
5. on agent: puppet agent -t
6. on master: puppet cert sign fqdn_of_agent

However, on puppetmaster1001 there are additional private keys in /var/lib/puppet/ssl/private_keys as well as the /var/lib/puppet/ssl/certs/puppet.pem cert, so we need to be more precise and move only the revoked cert aside. It's also both the agent and master in this case. So something like this (all commands on puppetmaster1001)

1. puppet agent --disable "regenerating cert"
2. cp -a /var/lib/puppet/ssl /var/lib/puppet/ssl.bak-$date
3. cp -a /var/lib/puppet/server /var/lib/puppet/server.bak-$date (paranoia)
4. mv /var/lib/puppet/ssl/certs/ca.pem /var/lib/puppet/ssl/certs/ca.pem.bak-$date
5. mv /var/lib/puppet/ssl/certs/puppetmaster1001.eqiad.wmnet.pem /var/lib/puppet/ssl/certs/puppetmaster1001.eqiad.wmnet.pem.bak-$date
6. mv /var/lib/puppet/ssl/private_keys/puppetmaster1001.eqiad.wmnet.pem /var/lib/puppet/ssl/private_keys/puppetmaster1001.eqiad.wmnet.pem.bak-$date
7. mv /var/lib/puppet/ssl/public_keys/puppetmaster1001.eqiad.wmnet.pem /var/lib/puppet/ssl/public_keys/puppetmaster1001.eqiad.wmnet.pem.bak-$date
8. puppet cert clean puppetmaster1001.eqiad.wmnet
9. puppet agent --enable
10. puppet agent -t
11. puppet cert sign puppetmaster1001.eqiad.wmnet

To reduce risk I think we should tackle this after depooling the eqiad puppet masters for upgrades