In order for etcd to be more up to our current goals, we want to do the following:
- Set up a MirrorMaker-like replica
- Switchover to codfw
- Re-configure the eqiad cluster to use the tls proxy
- Allow reading from the nearest datacenter (optional)
here are my ideas for this:
MirrorMaker-like replica
Temporary "emergency" switchover
We need to switch over to the codfw cluster because of a time-sensitive maintenance in eqiad, so we need to do as follows for now:
- Verify the replica from eqiad to codfw is currently correctly set up to '/conftool'
- Add a second SRV record for etcd.rw or something similar that can be used by conftool, so that all writes can be managed that way
- Reconfigure conftool to use it
- Reduce the TTL of all SRV records
- Switch the configurations of a few pybal hosts to use codfw, possibly just the backups; verify data after the restart are the same with the other element of the couple
- Switch the other clients (pybals included) by changing the SRV records for everything but conftool. verify it actively removes the connections from those hosts to eqiad
- Switch the record for conftool too
- Stop the etcd replication eqiad => codfw
- Re-raise the TTL of all SRV records
A longer-term plan
At the moment, we're interested only in replicating conftool data. I have given some thought about this and came to the conclusion that the best course of action is the following:
- Copy the data currently in /conftool in eqiad under /eqiad.wmnet/conftool by starting a replica
- Add to puppet a etcd_masterdc variable, and have the conftool_prefix hiera variable depend on that. This will make conftool/confd read/write to this new directory.
- Once puppet is ran everywhere, all reads/writes will go to the new
- replicate this tree to codfw 1:1
- On codfw, create /codfw.wmnet/conftool and replicate it 1:1 to eqiad
We will configure everything to just read/write to eqiad for now, and given conftool is not able to write to multiple clusters, this can seem not so useful, but it helps with our next goals, as we should see. Also, if we decide to change the way conftool works and allow multi-dc writes, we can benefit from this.
Allow reading from the nearest datacenter
Once we've defined etcd_masterdc in puppet, we will be able to make servers in the various DCs to read from the nearest available datacenter under the correct index, by simply changing the SRV records in the DNS. It would be wise to introduce different DNS records for reads and writes, so that conftool will always connect to the master. I think this could even come from discovery, but the level of etcdinception would make me uncomfortable. So, manual records for now!
Switchover to codfw
Whenever we want to switchover, the steps will be:
- set up a second, temporary local replication in codfw from /eqiad.wmnet/ to /codfw.wmnet
- Change the etcd_masterdc variable and run puppet everywhere it matters
- We stop the temporary replication
Re-configure the eqiad cluster to use TLS proxy
We will need to do the following:
- Ensure nothing reads from eqiad by changing DNS/other configs
- Prepare the new ECDSA certs, and commit them to puppet
- Disable auth (can be done via etcd-manage)
- Disable puppet across the config cluster in eqiad
- Switch the conf1* servers to use role::configcluster
- One machine at a time, stop etcd, run puppet, verify the server has reconnected to its current cluster.