We need to set up a second etcd cluster in codfw.
I would adopt the following tactics:
- Create a 3-node cluster on the conf2xxx servers already used for ZK
- Use an nginx proxy instead of the builtin etcd auth for additional scalability
- Replicate from the eqiad cluster with a process we can start/stop depending on which of the two clusters is the master.
Unfortunately the replication would be very easy to accomplish with etcd 3, as etcdctl has a "mirror maker" function that allows to do just that.
I'd stay short of suggesting we upgrade to etcd3 at the moment (although it might be a good idea at a later point).