**Background**
In order to reimage the eqiad conf* hosts (T428495), we first need to switch the etcd primary (R/W) cluster to codfw. This is what will allow us to fully depool etcd client traffic from eqiad in advance of the reimage.
**Procedure**
See https://wikitech.wikimedia.org/wiki/Etcd/Main_cluster#Master_cluster_switchover for prior art. A subsidiary goal of this work is to refresh that documentation.
1. Reduce the TTL for conftool (R/W) etcd client SRV records to 10 seconds.
[ ] https://gerrit.wikimedia.org/r/1319194
[ ] https://wikitech.wikimedia.org/wiki/DNS#Deploying_DNS_changes
2. (shortly before) Silence the EtcdReplicationDown alert.
3. Begin read-only in the etcd cluster we are switching from (eqiad).
[ ] https://gerrit.wikimedia.org/r/c/1319191
[ ] `sudo cumin A:conf-eqiad 'run-puppet-agent'`
4. Verify read-only
[ ] Attempting to depool a host should fail, e.g., `sudo confctl select 'name=wikikube-worker2330.codfw.wmnet' set/pooled=no`
5. Disable puppet on the current and new replication hosts
[ ] `sudo cumin 'conf2005.codfw.wmnet,conf1008.eqiad.wmnet' 'disable-puppet "etcd replication switchover"'`
6. Merge a puppet patch that disables etcd replication in the cluster we are switching to (codfw) and enables it in the cluster we are switching from (eqiad).
[ ] https://gerrit.wikimedia.org/r/1319192
7. Run puppet-agent on the current replication host in the cluster we are switching to (codfw) (replication stops)
[ ] (on conf2005.codfw.wmnet) `sudo run-puppet-agent -e "etcd replication switchover"`
[ ] (on conf2005.codfw.wmnet) confirm that `etcdmirror--eqiad-wmnet.service` has terminated
8. Switch the conftool (R/W) etcd client SRV records to the cluster we are switching to.
[ ] https://gerrit.wikimedia.org/r/1319195
[ ] https://wikitech.wikimedia.org/wiki/DNS#Deploying_DNS_changes
9. Set the replication index in the cluster we are switching from (eqiad)
[ ] Invoke index update script as in T433554#12169821 from any eqiad cluster member (e.g., the new replication host: conf1008)
10. Run puppet-agent on the new replication host in the cluster we are switching from (eqiad) (replication starts)
[ ] (on conf1008.eqiad.wmnet) `sudo run-puppet-agent -e "etcd replication switchover"`
[ ] (on conf1008.eqiad.wmnet) confirm that `etcdmirror--codfw-wmnet.service` has started
11. Test replication via a local mutation on any member of the cluster we are switching to.
[ ] (on conf1008.eqiad.wmnet) Monitor replication progress: `journalctl -f -u etcdmirror--codfw-wmnet.service`
[ ] (on conf2005.codfw.wmnet or any other codfw cluster member)
* `curl https://$(hostname -f):2379/v2/keys/test -XPUT -d value=""`
* `curl https://$(hostname -f):2379/v2/keys/test -XDELETE`
12. Begin read-write in the etcd cluster we are switching to (codfw)
[ ] https://gerrit.wikimedia.org/r/1319193
[ ] `sudo cumin A:conf-codfw 'run-puppet-agent'`
13. Verify read-write
[ ] Attempting to depool a host should succeed, e.g., `sudo confctl select 'name=wikikube-worker2330.codfw.wmnet' set/pooled=no`.
* **Remember to repool the host** (`sudo confctl select 'name=wikikube-worker2330.codfw.wmnet' set/pooled=yes`).
[ ] Announce all-clear,
14. Restore the TTL for conftool (R/W) etcd client SRV records to 5 minutes.
[ ] https://gerrit.wikimedia.org/r/1319196
[ ] https://wikitech.wikimedia.org/wiki/DNS#Deploying_DNS_changes