User and project data (files, databases) need to be replicated between the two primary datacenters for redundancy. This data has two sources: the NFS server and the Mysql databases.
After discussion within the ops team, the solution for filesystems that was picked is to have simple hourly replication between sites, and have the cross-site data be avaliable readonly in each location (as opposed to a shared filesystem that can be written to in both locations, which was considered too brittle). The codfw file server is already being set up with the capacity for thin snapshots for that purpose.
Databases will be replicated via the normal Mysql replication mechanism.