Status
As of ~ 13:50 UTC on 2026-08-04, codfw is the now the main-etcd primary cluster.
While no further operational actions are planned as part of this task, I will keep it open until I am able to refresh the Wikitech documentation.
Background
In order to reimage the eqiad conf* hosts (T428495), we first need to switch the etcd primary (R/W) cluster to codfw. This is what will allow us to fully depool etcd client traffic from eqiad in advance of the reimage.
Procedure
See https://wikitech.wikimedia.org/wiki/Etcd/Main_cluster#Master_cluster_switchover for prior art. A subsidiary goal of this work is to refresh that documentation.
Timing - Running through the full procedure below, the read-only period (steps 3 through 12) lasted ~ 17 minutes.
- Reduce the TTL for conftool (R/W) etcd client SRV records to 10 seconds.
- (shortly before) Silence the EtcdReplicationDown alert.
- Created c54ed5b4-f2c8-4797-8ab0-907a13fc9d93
- Begin read-only in the etcd cluster we are switching from (eqiad).
- https://gerrit.wikimedia.org/r/c/1319191
- sudo cumin A:conf-eqiad 'run-puppet-agent'
- Verify read-only
- Attempting to depool a host should fail, e.g., sudo confctl select 'name=wikikube-worker2330.codfw.wmnet' set/pooled=no
- Disable puppet on the current and new replication hosts
- sudo cumin 'conf2005.codfw.wmnet,conf1008.eqiad.wmnet' 'disable-puppet "etcd replication switchover"'
- Merge a puppet patch that disables etcd replication in the cluster we are switching to (codfw) and enables it in the cluster we are switching from (eqiad).
- Run puppet-agent on the current replication host in the cluster we are switching to (codfw) (replication stops)
- (on conf2005.codfw.wmnet) sudo run-puppet-agent -e "etcd replication switchover"
- (on conf2005.codfw.wmnet) confirm that etcdmirror--eqiad-wmnet.service has terminated
- Set the replication index in the cluster we are switching from (eqiad)
- Invoke index update script as in T433554#12169821 from any eqiad cluster member (e.g., the new replication host: conf1008)
- Run puppet-agent on the new replication host in the cluster we are switching from (eqiad) (replication starts)
- (on conf1008.eqiad.wmnet) sudo run-puppet-agent -e "etcd replication switchover"
- (on conf1008.eqiad.wmnet) confirm that etcdmirror--codfw-wmnet.service has started
- Test replication via a local mutation on any member of the cluster we are switching to.
- (on conf1008.eqiad.wmnet) Monitor replication progress: journalctl -f -u etcdmirror--codfw-wmnet.service
- (on conf2005.codfw.wmnet or any other codfw cluster member)
- curl https://$(hostname -f):2379/v2/keys/test -XPUT -d value=""
- curl https://$(hostname -f):2379/v2/keys/test -XDELETE
- Switch the conftool (R/W) etcd client SRV records to the cluster we are switching to.
- Begin read-write in the etcd cluster we are switching to (codfw)
- https://gerrit.wikimedia.org/r/1319193
- sudo cumin A:conf-codfw 'run-puppet-agent'
- Verify read-write
- Attempting to depool a host should succeed, e.g., sudo confctl select 'name=wikikube-worker2330.codfw.wmnet' set/pooled=no.
- Remember to repool the host (e.g., sudo confctl select 'name=wikikube-worker2330.codfw.wmnet' set/pooled=yes).
- Attempting to depool a host should succeed, e.g., sudo confctl select 'name=wikikube-worker2330.codfw.wmnet' set/pooled=no.
- Restart hiddenparma (requestctl), since it will likely have cached connections toward the cluster we are switching from (i.e., the next edit attempt will fail).
- (alert1002.wikimedia.org, alert2002.wikimedia.org) sudo systemctl restart hiddenparma.service
- Announce all-clear
- Delete EtcdReplicationDown silence. If the alert is still firing in the cluster we are switching to (i.e., where etcd-mirror was running previously), it means the ops prometheus host(s) in that DC need a puppet-agent run (you can just wait for this to happen organically before deleting the silence).
- Deleted c54ed5b4-f2c8-4797-8ab0-907a13fc9d93
- Restore the TTL for conftool (R/W) etcd client SRV records to 5 minutes.