Steps to reproduce
- Create a new instance in tools to try to pool to be k8s worker node (https://wikitech.wikimedia.org/wiki/Tools_Kubernetes#Worker_nodes)
- Ssh in
- Start preparing to switch puppetmasters: 'sudo rm -rf /var/lib/puppet/ssl'
- Try to run puppet
- Watch as it times out
- Try to curl any port on the puppetmaster: 'curl tools-puppetmaster-01:3422', watch as it times out (rather than produce connection failure immediately)
- Wait for a while (10-20mins)
- Everyting is ok now.