Before we start integrating conf200[7-9], we'll need to switch the etcd primary cluster back to eqiad. This will give us the ability to depool client traffic from codfw while work is ongoing.
This will use the procedure in https://wikitech.wikimedia.org/wiki/Etcd/Main_cluster#Primary_cluster_switchover.
Primary time slot: Wednesday 19th August 13:30 UTC
Timing: ~ 12m from the nginx reloads that entered read-only to the reloads that exited it.
Preparation
- Reduce the TTL for conftool (R/W) etcd client SRV records to 10 seconds
- https://gerrit.wikimedia.org/r/1326363
- Silence the EtcdReplicationDown alert shortly before work begins.
- e7b939b5-67fa-4eee-9f12-891c47f84981
Switchover
- Begin read-only in the etcd cluster we are switching from. Disruption begins.
- https://gerrit.wikimedia.org/r/1326355
- sudo cumin A:conf-codfw 'run-puppet-agent'
- Verify read-only
- Attempting to depool a suitable host should fail (e.g., sudo confctl select "name=$FDQN" set/pooled=no).
- Disable puppet on the current and new replication hosts
- sudo cumin 'conf2005.codfw.wmnet,conf1008.eqiad.wmnet' 'disable-puppet "etcd replication switchover"'
- Merge a puppet patch that disables etcd replication in the cluster we are switching to and enables it in the cluster we are switching from (i.e., affecting the two hosts on which you just disabled puppet).
- https://gerrit.wikimedia.org/r/1326356
- Run puppet-agent on the current replication host in the cluster we are switching to (replication stops)
- (on conf1008.eqiad.wmnet) sudo run-puppet-agent -e "etcd replication switchover"
- (on conf1008.eqiad.wmnet) confirm that etcdmirror--codfw-wmnet.service has terminated (this may take up to 60 seconds)
- Set the replication index in the cluster we are switching from.
- Use the python script in P95724, which can be invoked from any member of the cluster we're switching from (e.g., the new replication host). Note that you will be prompted to approve the etcd write.
- (on conf2005.codfw.wmnet) python3 set_mirror_index.py --protocol https --old-replica conf1008.eqiad.wmnet:4001 --new-replica $(hostname -f):2379 --prefix __ROOT__
- Run puppet-agent on the new replication host in the cluster we are switching from (replication starts).
- (on conf2005.codfw.wmnet) sudo run-puppet-agent -e "etcd replication switchover"
- (on conf2005.codfw.wmnet) confirm that etcdmirror--eqiad-wmnet.service has started
- Test replication via a local mutation on any member of the cluster we are switching to.
- (on conf2005.codfw.wmnet) Monitor replication progress: journalctl -f -u etcdmirror--eqiad-wmnet.service
- (on conf1008.eqiad.wmnet or any other eqiad cluster member) Write (and then delete) a suitable test key:
- curl https://$(hostname -f):2379/v2/keys/test -XPUT -d value=""
- curl https://$(hostname -f):2379/v2/keys/test -XDELETE
- Switch the conftool (R/W) etcd client SRV records to the cluster we are switching to.
- https://gerrit.wikimedia.org/r/1326364
- Begin read-write in the etcd cluster we are switching to.
- https://gerrit.wikimedia.org/r/1326357
- sudo cumin A:conf-eqiad 'run-puppet-agent'
- Verify read-write
- Attempting to depool a suitable host should succeed, e.g., sudo confctl select "name=$FQDN" set/pooled=no.
- Remember to repool the host (e.g., sudo confctl select "name=$FQDN" set/pooled=yes).
- Restart hiddenparma (Requestctl), since it will have cached connections toward the cluster we are switching from, which is now read-only - i.e., subsequent edits will fail.
- (on alert1002.wikimedia.org, alert2002.wikimedia.org) sudo systemctl restart hiddenparma.service
- Restart additional read-only services that happen to track the conftool-specific SRV records. Since these are read-only, this is far less urgent than anything above (i.e., they will continue to function even if still connected to the cluster we have switched from).
- Conftool2git - sudo systemctl restart conftool2git.service on the profile::conftool2git::active_host
- MariaDB/Zarcillo - A service running in the aux-k8s-eqiad cluster. Coordinate with Data Persistence (i.e., ask them to restart it).
Cleanup
- Delete the EtcdReplicationDown silence.
- If the alert is still firing in the cluster we are switching to (i.e., where etcd-mirror was running previously), it means the ops prometheus host(s) in that DC need a puppet-agent run (you can just wait for this to happen organically before deleting the silence).
- Restore the TTL for conftool (R/W) etcd client SRV records to 5 minutes.
- https://gerrit.wikimedia.org/r/1326365