At this time, various stakeholders agreed upon a general method of making this all work in T267376.
With this patch https://gerrit.wikimedia.org/r/627379, the existing haproxy servers have the ability to serve the multi-instance replicas but for the firewall and other requirements detailed in T267376 (to be added here). Things still to work out:
[x] Establish the correct route through firewalls to the clouddb-wikireplica-proxy VMs in the cloud, per the agreed on structure
[] Write detailed documentation for WMCS and Data Persistence for depooling and related operations on the proxy level
[x] Determine if dbproxy1018 and dbproxy1019 should have auto failover set up for the multi-instance replicas. Right now they don't without an override, and that seems incorrect. Each port just sends to the main backend. A puppet change should fix that.
[x] Create the DNS names needed and adjust the wmcs-wikireplicas-dns script accordingly
Generally agreed on routing structure:
{F33946999}
The main diff compared to the previous architecture diagram is that we introduce another proxy layer inside the CloudVPS virtual network. This `proxy layer 1` is able to abstract away database sections for customers.
Then `proxy layer 2` receives connections from the first layer, using the general egress CloudVPS NAT public IPv4 address, and is responsible to proxy again to the actual database, using different TCP port selectors which is only known between `proxy layer 1` and `proxy layer 2`.
The flow is something like this:
```
client
->
enwiki.web.svc.clouddbservices.eqiad1.wikimedia.cloud IN CNAME s1.web.svc.clouddbservices.eqiad1.wikimedia.cloud
->
s1.web.svc,clouddbservices.eqiad1.wikimedia.cloud (proxy layer 1 -- 172.16.x.x -- TCP/3306)
->
dbproxy1019 (proxy layer 2 -- web db address (208.80.x.x) -- s1 port TCP/XXXX)
->
clouddb1013 or any other db mysql
client
->
enwiki.analytics.svc.clouddbservices.eqiad1.wikimedia.cloud IN CNAME s1.analytics.svc.clouddbservices.eqiad1.wikimedia.cloud
->
s1.analytics.svc,clouddbservices.eqiad1.wikimedia.cloud (proxy layer 1 -- 172.16.y.y -- TCP/3306)
->
dbproxy1018 (proxy layer 2 -- analytics db address (208.80.y.y) -- s1 port TCP/YYYY)
->
clouddb1013 or any other db mysql
```