Change Details

**Background** In order to reimage the eqiad conf* hosts (T428495), we first need to switch the etcd primary (R/W) cluster to codfw. This is what will allow us to fully depool etcd client traffic from eqiad in advance of the reimage. **Procedure** See https://wikitech.wikimedia.org/wiki/Etcd/Main_cluster#Master_cluster_switchover for prior art. A subsidiary goal of this work is to refresh that documentation. 1. Reduce the TTL for conftool (R/W) etcd client SRV records to 10 seconds. [x] https://gerrit.wikimedia.org/r/1319194 [x] https://wikitech.wikimedia.org/wiki/DNS#Deploying_DNS_changes 2. (shortly before) Silence the EtcdReplicationDown alert. [ ] (ID) 3. Begin read-only in the etcd cluster we are switching from (eqiad). [ ] https://gerrit.wikimedia.org/r/c/1319191 [ ] `sudo cumin A:conf-eqiad 'run-puppet-agent'` 4. Verify read-only [ ] Attempting to depool a host should fail, e.g., `sudo confctl select 'name=wikikube-worker2330.codfw.wmnet' set/pooled=no` 5. Disable puppet on the current and new replication hosts [ ] `sudo cumin 'conf2005.codfw.wmnet,conf1008.eqiad.wmnet' 'disable-puppet "etcd replication switchover"'` 6. Merge a puppet patch that disables etcd replication in the cluster we are switching to (codfw) and enables it in the cluster we are switching from (eqiad). [ ] https://gerrit.wikimedia.org/r/1319192 7. Run puppet-agent on the current replication host in the cluster we are switching to (codfw) (replication stops) [ ] (on conf2005.codfw.wmnet) `sudo run-puppet-agent -e "etcd replication switchover"` [ ] (on conf2005.codfw.wmnet) confirm that `etcdmirror--eqiad-wmnet.service` has terminated 8. Switch the conftool (R/W) etcd client SRV records to the cluster we are switching to. [ ] https://gerrit.wikimedia.org/r/1319195 [ ] https://wikitech.wikimedia.org/wiki/DNS#Deploying_DNS_changes 9. Set the replication index in the cluster we are switching from (eqiad) [ ] Invoke index update script as in T433554#12169821 from any eqiad cluster member (e.g., the new replication host: conf1008) 10. Run puppet-agent on the new replication host in the cluster we are switching from (eqiad) (replication starts) [ ] (on conf1008.eqiad.wmnet) `sudo run-puppet-agent -e "etcd replication switchover"` [ ] (on conf1008.eqiad.wmnet) confirm that `etcdmirror--codfw-wmnet.service` has started 11. Test replication via a local mutation on any member of the cluster we are switching to. [ ] (on conf1008.eqiad.wmnet) Monitor replication progress: `journalctl -f -u etcdmirror--codfw-wmnet.service` [ ] (on conf2005.codfw.wmnet or any other codfw cluster member) * `curl https://$(hostname -f):2379/v2/keys/test -XPUT -d value=""` * `curl https://$(hostname -f):2379/v2/keys/test -XDELETE` 12. Begin read-write in the etcd cluster we are switching to (codfw) [ ] https://gerrit.wikimedia.org/r/1319193 [ ] `sudo cumin A:conf-codfw 'run-puppet-agent'` 13. Verify read-write [ ] Attempting to depool a host should succeed, e.g., `sudo confctl select 'name=wikikube-worker2330.codfw.wmnet' set/pooled=no`. * **Remember to repool the host** (`sudo confctl select 'name=wikikube-worker2330.codfw.wmnet' set/pooled=yes`). [ ] Announce all-clear, 14. Restore the TTL for conftool (R/W) etcd client SRV records to 5 minutes. [ ] https://gerrit.wikimedia.org/r/1319196 [ ] https://wikitech.wikimedia.org/wiki/DNS#Deploying_DNS_changes

**Background** In order to reimage the eqiad conf* hosts (T428495), we first need to switch the etcd primary (R/W) cluster to codfw. This is what will allow us to fully depool etcd client traffic from eqiad in advance of the reimage. **Procedure** See https://wikitech.wikimedia.org/wiki/Etcd/Main_cluster#Master_cluster_switchover for prior art. A subsidiary goal of this work is to refresh that documentation. 1. Reduce the TTL for conftool (R/W) etcd client SRV records to 10 seconds. [x] https://gerrit.wikimedia.org/r/1319194 [x] https://wikitech.wikimedia.org/wiki/DNS#Deploying_DNS_changes 2. (shortly before) Silence the EtcdReplicationDown alert. [x] `c54ed5b4-f2c8-4797-8ab0-907a13fc9d93` 3. Begin read-only in the etcd cluster we are switching from (eqiad). [ ] https://gerrit.wikimedia.org/r/c/1319191 [ ] `sudo cumin A:conf-eqiad 'run-puppet-agent'` 4. Verify read-only [ ] Attempting to depool a host should fail, e.g., `sudo confctl select 'name=wikikube-worker2330.codfw.wmnet' set/pooled=no` 5. Disable puppet on the current and new replication hosts [ ] `sudo cumin 'conf2005.codfw.wmnet,conf1008.eqiad.wmnet' 'disable-puppet "etcd replication switchover"'` 6. Merge a puppet patch that disables etcd replication in the cluster we are switching to (codfw) and enables it in the cluster we are switching from (eqiad). [ ] https://gerrit.wikimedia.org/r/1319192 7. Run puppet-agent on the current replication host in the cluster we are switching to (codfw) (replication stops) [ ] (on conf2005.codfw.wmnet) `sudo run-puppet-agent -e "etcd replication switchover"` [ ] (on conf2005.codfw.wmnet) confirm that `etcdmirror--eqiad-wmnet.service` has terminated 8. Switch the conftool (R/W) etcd client SRV records to the cluster we are switching to. [ ] https://gerrit.wikimedia.org/r/1319195 [ ] https://wikitech.wikimedia.org/wiki/DNS#Deploying_DNS_changes 9. Set the replication index in the cluster we are switching from (eqiad) [ ] Invoke index update script as in T433554#12169821 from any eqiad cluster member (e.g., the new replication host: conf1008) 10. Run puppet-agent on the new replication host in the cluster we are switching from (eqiad) (replication starts) [ ] (on conf1008.eqiad.wmnet) `sudo run-puppet-agent -e "etcd replication switchover"` [ ] (on conf1008.eqiad.wmnet) confirm that `etcdmirror--codfw-wmnet.service` has started 11. Test replication via a local mutation on any member of the cluster we are switching to. [ ] (on conf1008.eqiad.wmnet) Monitor replication progress: `journalctl -f -u etcdmirror--codfw-wmnet.service` [ ] (on conf2005.codfw.wmnet or any other codfw cluster member) * `curl https://$(hostname -f):2379/v2/keys/test -XPUT -d value=""` * `curl https://$(hostname -f):2379/v2/keys/test -XDELETE` 12. Begin read-write in the etcd cluster we are switching to (codfw) [ ] https://gerrit.wikimedia.org/r/1319193 [ ] `sudo cumin A:conf-codfw 'run-puppet-agent'` 13. Verify read-write [ ] Attempting to depool a host should succeed, e.g., `sudo confctl select 'name=wikikube-worker2330.codfw.wmnet' set/pooled=no`. * **Remember to repool the host** (`sudo confctl select 'name=wikikube-worker2330.codfw.wmnet' set/pooled=yes`). [ ] Announce all-clear, 14. Restore the TTL for conftool (R/W) etcd client SRV records to 5 minutes. [ ] https://gerrit.wikimedia.org/r/1319196 [ ] https://wikitech.wikimedia.org/wiki/DNS#Deploying_DNS_changes

**Background** In order to reimage the eqiad conf* hosts (T428495), we first need to switch the etcd primary (R/W) cluster to codfw. This is what will allow us to fully depool etcd client traffic from eqiad in advance of the reimage. **Procedure** See https://wikitech.wikimedia.org/wiki/Etcd/Main_cluster#Master_cluster_switchover for prior art. A subsidiary goal of this work is to refresh that documentation. 1. Reduce the TTL for conftool (R/W) etcd client SRV records to 10 seconds. [x] https://gerrit.wikimedia.org/r/1319194 [x] https://wikitech.wikimedia.org/wiki/DNS#Deploying_DNS_changes 2. (shortly before) Silence the EtcdReplicationDown alert. [ ] (ID)x] `c54ed5b4-f2c8-4797-8ab0-907a13fc9d93` 3. Begin read-only in the etcd cluster we are switching from (eqiad). [ ] https://gerrit.wikimedia.org/r/c/1319191 [ ] `sudo cumin A:conf-eqiad 'run-puppet-agent'` 4. Verify read-only [ ] Attempting to depool a host should fail, e.g., `sudo confctl select 'name=wikikube-worker2330.codfw.wmnet' set/pooled=no` 5. Disable puppet on the current and new replication hosts [ ] `sudo cumin 'conf2005.codfw.wmnet,conf1008.eqiad.wmnet' 'disable-puppet "etcd replication switchover"'` 6. Merge a puppet patch that disables etcd replication in the cluster we are switching to (codfw) and enables it in the cluster we are switching from (eqiad). [ ] https://gerrit.wikimedia.org/r/1319192 7. Run puppet-agent on the current replication host in the cluster we are switching to (codfw) (replication stops) [ ] (on conf2005.codfw.wmnet) `sudo run-puppet-agent -e "etcd replication switchover"` [ ] (on conf2005.codfw.wmnet) confirm that `etcdmirror--eqiad-wmnet.service` has terminated 8. Switch the conftool (R/W) etcd client SRV records to the cluster we are switching to. [ ] https://gerrit.wikimedia.org/r/1319195 [ ] https://wikitech.wikimedia.org/wiki/DNS#Deploying_DNS_changes 9. Set the replication index in the cluster we are switching from (eqiad) [ ] Invoke index update script as in T433554#12169821 from any eqiad cluster member (e.g., the new replication host: conf1008) 10. Run puppet-agent on the new replication host in the cluster we are switching from (eqiad) (replication starts) [ ] (on conf1008.eqiad.wmnet) `sudo run-puppet-agent -e "etcd replication switchover"` [ ] (on conf1008.eqiad.wmnet) confirm that `etcdmirror--codfw-wmnet.service` has started 11. Test replication via a local mutation on any member of the cluster we are switching to. [ ] (on conf1008.eqiad.wmnet) Monitor replication progress: `journalctl -f -u etcdmirror--codfw-wmnet.service` [ ] (on conf2005.codfw.wmnet or any other codfw cluster member) * `curl https://$(hostname -f):2379/v2/keys/test -XPUT -d value=""` * `curl https://$(hostname -f):2379/v2/keys/test -XDELETE` 12. Begin read-write in the etcd cluster we are switching to (codfw) [ ] https://gerrit.wikimedia.org/r/1319193 [ ] `sudo cumin A:conf-codfw 'run-puppet-agent'` 13. Verify read-write [ ] Attempting to depool a host should succeed, e.g., `sudo confctl select 'name=wikikube-worker2330.codfw.wmnet' set/pooled=no`. * **Remember to repool the host** (`sudo confctl select 'name=wikikube-worker2330.codfw.wmnet' set/pooled=yes`). [ ] Announce all-clear, 14. Restore the TTL for conftool (R/W) etcd client SRV records to 5 minutes. [ ] https://gerrit.wikimedia.org/r/1319196 [ ] https://wikitech.wikimedia.org/wiki/DNS#Deploying_DNS_changes