We are maintaining two datacenters with the goal of surviving a DC-wide outage, such as a power failure or fiber cut. Supporting multi-DC replication has been a design consideration for RESTBase from the start. By choosing Cassandra as the storage backend, we get to use Cassandra's relatively mature cross-DC replication support.
We are in the process of purchasing a replica cluster for codfw (see T93790). The hardware there will hopefully come online before the end of this quarter. In the meantime, we should prepare and test cross-DC replication.
We don't have a general ipsec setup between the datacenters, so we'll likely need to [encrypt and strongly authenticate](http://docs.datastax.com/en/cassandra/2.1/cassandra/security/secureSSLCertificates_t.html) the cross-DC connections at the Cassandra level. Assuming one instance per hardware node (depends on T95253), testing at the full replication volume might require six nodes to keep up with compaction. Unless there are that many spares in codfw we might not be able to test this fully with the production cluster. We could however consider setting this up for the staging cluster, which has modest and controllable resource needs. Any set of three nodes (SSD or not) in codfw should be sufficient to test this in staging.