Change Details

We had a bit of a mess in Tools today, that went like this: 1) I switched tools nodes to a new puppetmaster 2) puppet certs were updated on tools VMs 3) we rebooted a few etcd nodes, causing them to load the new puppet certs 4) now etcd clients and masters disagreed about certs and broke The dangerous period between 2) and 3) should be avoided -- the puppet config for etcd should depend on cert changes so that the service restarts immediately.

We had a bit of a mess in Tools today, that went like this: 1) I switched tools nodes to a new puppetmaster 2) puppet certs were updated on tools VMs 3) we restarted the k8s components on the k8s-master, and they tried to validate the local puppet cert and check if it matched with the certs etcd presented 4) the etcd cluster was still using the old certs, which caused kube-apiserver to fail with etcd misconfigured errors 5) we restarted etcd on all the etcd clients, they picked up the new certs and then thinks got better The puppet config for etcd should depend on cert changes so that the etcd service restarts immediately.

We had a bit of a mess in Tools today, that went like this: 1) I switched tools nodes to a new puppetmaster 2) puppet certs were updated on tools VMs 3) we reboostarted a few etcd nodesthe k8s components on the k8s-master, causing themand they tried to loadvalidate the newlocal puppetet cert and check if it matched with the certsts etcd presented 4) now etcd clients and masters disagreed about certs and brokethe etcd cluster was still using the old certs, which caused kube-apiserver to fail with etcd misconfigured errors 5) we restarted etcd on all the etcd clients, they picked up the new certs and then thinks got better The dangerous period between 2) and 3) should be avoided -- the puppet config for etcd should depend on cert changes so that the etcd service restarts immediately.