Change Details

In order for etcd to be more up to our current goals, we want to do the following: [] Set up a MirrorMaker-like replica [] Switchover to codfw [] Re-configure the eqiad cluster to use the tls proxy [] Allow reading from the nearest datacenter (optional) here are my ideas for this: == MirrorMaker-like replica === Temporary "emergency" switchover We need to switch over to the codfw cluster because of a time-sensitive maintenance in eqiad, so we need to do as follows for now: [] Verify the replica from eqiad to codfw is currently correctly set up to '/conftool' [] Add a second SRV record for `etcd.rw` or something similar that can be used by conftool, so that all writes can be managed that way [] Reconfigure conftool to use it [] Reduce the TTL of all SRV records [] Switch the configurations of a few pybal hosts to use codfw, possibly just the backups; verify data after the restart are the same with the other element of the couple [] Switch the other clients (pybals included) by changing the SRV records for everything but conftool. verify it actively removes the connections from those hosts to eqiad [] Switch the record for conftool too [] Stop the etcd replication `eqiad => codfw` [] Re-raise the TTL of all SRV records === A longer-term plan At the moment, we're interested only in replicating conftool data. I have given some thought about this and came to the conclusion that the best course of action is the following: # Copy the data currently in /conftool in eqiad under /eqiad.wmnet/conftool by starting a replica # Add to puppet a `etcd_masterdc` variable, and have the `conftool_prefix` hiera variable depend on that. This will make conftool/confd read/write to this new directory. # Once puppet is ran everywhere, all reads/writes will go to the new # replicate this tree to codfw 1:1 # On codfw, create `/codfw.wmnet/conftool` and replicate it 1:1 to eqiad We will configure everything to just read/write to eqiad for now, and given conftool is not able to write to multiple clusters, this can seem not so useful, but it helps with our next goals, as we should see. Also, if we decide to change the way conftool works and allow multi-dc writes, we can benefit from this. == Allow reading from the nearest datacenter Once we've defined `etcd_masterdc` in puppet, we will be able to make servers in the various DCs to read from the nearest available datacenter under the correct index, by simply changing the SRV records in the DNS. It would be wise to introduce different DNS records for reads and writes, so that conftool will always connect to the master. I think this could even come from discovery, but the level of etcdinception would make me uncomfortable. So, manual records for now! == Switchover to codfw Whenever we want to switchover, the steps will be: # set up a second, temporary local replication in codfw from `/eqiad.wmnet/` to `/codfw.wmnet` # Change the `etcd_masterdc` variable and run puppet everywhere it matters # We stop the temporary replication === Re-configure the eqiad cluster to use TLS proxy We will need to do the following: # Ensure nothing reads from eqiad by changing DNS/other configs # Prepare the new ECDSA certs, and commit them to puppet # Disable auth (can be done via etcd-manage) # Disable puppet across the config cluster in eqiad # Switch the conf1* servers to use role::configcluster # One machine at a time, stop etcd, run puppet, verify the server has reconnected to its current cluster.

In order for etcd to be more up to our current goals, we want to do the following: [] Set up a MirrorMaker-like replica [] Switchover to codfw [] Re-configure the eqiad cluster to use the tls proxy [] Allow reading from the nearest datacenter (optional) here are my ideas for this: == MirrorMaker-like replica === Temporary "emergency" switchover We need to switch over to the codfw cluster because of a time-sensitive maintenance in eqiad, so we need to do as follows for now: [x] Verify the replica from eqiad to codfw is currently correctly set up to '/conftool' [] Add a second SRV record for `etcd.rw` or something similar that can be used by conftool, so that all writes can be managed that way [] Reconfigure conftool to use it [] Reduce the TTL of all SRV records [] Switch the configurations of a few pybal hosts to use codfw, possibly just the backups; verify data after the restart are the same with the other element of the couple [] Switch the other clients (pybals included) by changing the SRV records for everything but conftool. verify it actively removes the connections from those hosts to eqiad [] Switch the record for conftool too [] Stop the etcd replication `eqiad => codfw` [] Re-raise the TTL of all SRV records === A longer-term plan At the moment, we're interested only in replicating conftool data. I have given some thought about this and came to the conclusion that the best course of action is the following: # Copy the data currently in /conftool in eqiad under /eqiad.wmnet/conftool by starting a replica # Add to puppet a `etcd_masterdc` variable, and have the `conftool_prefix` hiera variable depend on that. This will make conftool/confd read/write to this new directory. # Once puppet is ran everywhere, all reads/writes will go to the new # replicate this tree to codfw 1:1 # On codfw, create `/codfw.wmnet/conftool` and replicate it 1:1 to eqiad We will configure everything to just read/write to eqiad for now, and given conftool is not able to write to multiple clusters, this can seem not so useful, but it helps with our next goals, as we should see. Also, if we decide to change the way conftool works and allow multi-dc writes, we can benefit from this. == Allow reading from the nearest datacenter Once we've defined `etcd_masterdc` in puppet, we will be able to make servers in the various DCs to read from the nearest available datacenter under the correct index, by simply changing the SRV records in the DNS. It would be wise to introduce different DNS records for reads and writes, so that conftool will always connect to the master. I think this could even come from discovery, but the level of etcdinception would make me uncomfortable. So, manual records for now! == Switchover to codfw Whenever we want to switchover, the steps will be: # set up a second, temporary local replication in codfw from `/eqiad.wmnet/` to `/codfw.wmnet` # Change the `etcd_masterdc` variable and run puppet everywhere it matters # We stop the temporary replication === Re-configure the eqiad cluster to use TLS proxy We will need to do the following: # Ensure nothing reads from eqiad by changing DNS/other configs # Prepare the new ECDSA certs, and commit them to puppet # Disable auth (can be done via etcd-manage) # Disable puppet across the config cluster in eqiad # Switch the conf1* servers to use role::configcluster # One machine at a time, stop etcd, run puppet, verify the server has reconnected to its current cluster.

In order for etcd to be more up to our current goals, we want to do the following: [] Set up a MirrorMaker-like replica [] Switchover to codfw [] Re-configure the eqiad cluster to use the tls proxy [] Allow reading from the nearest datacenter (optional) here are my ideas for this: == MirrorMaker-like replica === Temporary "emergency" switchover We need to switch over to the codfw cluster because of a time-sensitive maintenance in eqiad, so we need to do as follows for now: [x] Verify the replica from eqiad to codfw is currently correctly set up to '/conftool' [] Add a second SRV record for `etcd.rw` or something similar that can be used by conftool, so that all writes can be managed that way [] Reconfigure conftool to use it [] Reduce the TTL of all SRV records [] Switch the configurations of a few pybal hosts to use codfw, possibly just the backups; verify data after the restart are the same with the other element of the couple [] Switch the other clients (pybals included) by changing the SRV records for everything but conftool. verify it actively removes the connections from those hosts to eqiad [] Switch the record for conftool too [] Stop the etcd replication `eqiad => codfw` [] Re-raise the TTL of all SRV records === A longer-term plan At the moment, we're interested only in replicating conftool data. I have given some thought about this and came to the conclusion that the best course of action is the following: # Copy the data currently in /conftool in eqiad under /eqiad.wmnet/conftool by starting a replica # Add to puppet a `etcd_masterdc` variable, and have the `conftool_prefix` hiera variable depend on that. This will make conftool/confd read/write to this new directory. # Once puppet is ran everywhere, all reads/writes will go to the new # replicate this tree to codfw 1:1 # On codfw, create `/codfw.wmnet/conftool` and replicate it 1:1 to eqiad We will configure everything to just read/write to eqiad for now, and given conftool is not able to write to multiple clusters, this can seem not so useful, but it helps with our next goals, as we should see. Also, if we decide to change the way conftool works and allow multi-dc writes, we can benefit from this. == Allow reading from the nearest datacenter Once we've defined `etcd_masterdc` in puppet, we will be able to make servers in the various DCs to read from the nearest available datacenter under the correct index, by simply changing the SRV records in the DNS. It would be wise to introduce different DNS records for reads and writes, so that conftool will always connect to the master. I think this could even come from discovery, but the level of etcdinception would make me uncomfortable. So, manual records for now! == Switchover to codfw Whenever we want to switchover, the steps will be: # set up a second, temporary local replication in codfw from `/eqiad.wmnet/` to `/codfw.wmnet` # Change the `etcd_masterdc` variable and run puppet everywhere it matters # We stop the temporary replication === Re-configure the eqiad cluster to use TLS proxy We will need to do the following: # Ensure nothing reads from eqiad by changing DNS/other configs # Prepare the new ECDSA certs, and commit them to puppet # Disable auth (can be done via etcd-manage) # Disable puppet across the config cluster in eqiad # Switch the conf1* servers to use role::configcluster # One machine at a time, stop etcd, run puppet, verify the server has reconnected to its current cluster.