**Background**
In order to reimage the eqiad conf* hosts (T428495), we first need to switch the etcd primary (R/W) cluster to codfw. This is what will allow us to fully depool etcd client traffic from eqiad in advance of the reimage.
**Procedure**
See https://wikitech.wikimedia.org/wiki/Etcd/Main_cluster#Master_cluster_switchover for prior art. A subsidiary goal of this work is to refresh that documentation.
1. Reduce the TTL for conftool (R/W) etcd client SRV records to 10 seconds.
[x] https://gerrit.wikimedia.org/r/1319194
[x] https://wikitech.wikimedia.org/wiki/DNS#Deploying_DNS_changes
2. (shortly before) Silence the EtcdReplicationDown alert.
[x] Created `c54ed5b4-f2c8-4797-8ab0-907a13fc9d93`
3. Begin read-only in the etcd cluster we are switching from (eqiad).
[x] https://gerrit.wikimedia.org/r/c/1319191
[x] `sudo cumin A:conf-eqiad 'run-puppet-agent'`
4. Verify read-only
[x] Attempting to depool a host should fail, e.g., `sudo confctl select 'name=wikikube-worker2330.codfw.wmnet' set/pooled=no`
5. Disable puppet on the current and new replication hosts
[x] `sudo cumin 'conf2005.codfw.wmnet,conf1008.eqiad.wmnet' 'disable-puppet "etcd replication switchover"'`
6. Merge a puppet patch that disables etcd replication in the cluster we are switching to (codfw) and enables it in the cluster we are switching from (eqiad).
[x] https://gerrit.wikimedia.org/r/1319192
7. Run puppet-agent on the current replication host in the cluster we are switching to (codfw) (replication stops)
[x] (on conf2005.codfw.wmnet) `sudo run-puppet-agent -e "etcd replication switchover"`
[x] (on conf2005.codfw.wmnet) confirm that `etcdmirror--eqiad-wmnet.service` has terminated
8. Switch the conftool (R/W) etcd client SRV records to the cluster we are switching to. -- TODO: Move this down.
[x] https://gerrit.wikimedia.org/r/1319195
[x] https://wikitech.wikimedia.org/wiki/DNS#Deploying_DNS_changes
9. Set the replication index in the cluster we are switching from (eqiad)
[x] Invoke index update script as in T433554#12169821 from any eqiad cluster member (e.g., the new replication host: conf1008)
10. Run puppet-agent on the new replication host in the cluster we are switching from (eqiad) (replication starts)
[x] (on conf1008.eqiad.wmnet) `sudo run-puppet-agent -e "etcd replication switchover"`
[x] (on conf1008.eqiad.wmnet) confirm that `etcdmirror--codfw-wmnet.service` has started
11. Test replication via a local mutation on any member of the cluster we are switching to.
[x] (on conf1008.eqiad.wmnet) Monitor replication progress: `journalctl -f -u etcdmirror--codfw-wmnet.service`
[x] (on conf2005.codfw.wmnet or any other codfw cluster member)
* `curl https://$(hostname -f):2379/v2/keys/test -XPUT -d value=""`
* `curl https://$(hostname -f):2379/v2/keys/test -XDELETE`
12. Begin read-write in the etcd cluster we are switching to (codfw)
[x] https://gerrit.wikimedia.org/r/1319193
[x] `sudo cumin A:conf-codfw 'run-puppet-agent'`
13. Verify read-write
[x] Attempting to depool a host should succeed, e.g., `sudo confctl select 'name=wikikube-worker2330.codfw.wmnet' set/pooled=no`.
* **Remember to repool the host** (`sudo confctl select 'name=wikikube-worker2330.codfw.wmnet' set/pooled=yes`).
14. Restart hiddenparma
[x] (alert1002.wikimedia.org, alert2002.wikimedia.org) `sudo systemctl restart hiddenparma.service`
15. Announce all-clear
16. Delete EtcdReplicationDown silence. If the alert is still firing in the cluster we are switching to, it means the ops prometheus host(s) in that DC need a puppet-agent run (you can just wait for this to happen organically).
[x] Announce all-clear,x] Deleted `c54ed5b4-f2c8-4797-8ab0-907a13fc9d93`
1417. Restore the TTL for conftool (R/W) etcd client SRV records to 5 minutes.
[ ] https://gerrit.wikimedia.org/r/1319196
[ ] https://wikitech.wikimedia.org/wiki/DNS#Deploying_DNS_changes