We had a bit of a mess in Tools today, that went like this:
- I switched tools nodes to a new puppetmaster
- puppet certs were updated on tools VMs
- we restarted the k8s components on the k8s-master, and they tried to validate the local puppet cert and check if it matched with the certs etcd presented
- the etcd cluster was still using the old certs, which caused kube-apiserver to fail with etcd misconfigured errors
- we restarted etcd on all the etcd clients, they picked up the new certs and then thinks got better
The puppet config for etcd should depend on cert changes so that the etcd service restarts immediately.