Thanks @arturo, I think that sums up the options we discussed. A few notes:

Existing Production LVS

One other option that was mentioned, and I'm not sure how viable it is, would be to utilize the existing production LVS. But you made a good point that this doesn't align with the goal of keeping cloud services as a separate realm.

Anycast

The last option you list is basically Anycasting the traffic from the cloudsw across a bunch of servers. This works well, the switch will forward each flow (based on src+dst IP) to one of the servers. If a server dies the BGP session will too, and the switch will lose that route and redirect traffic to the remaining servers. You can also add additional health-checks to force the BGP down on a server if there is a more subtle problem with a service that doesn't bring the box itself down.

Anycasting is usually stateless on the switch. One problem that causes is that all flows get remapped if the number of available back-ends change, but often that's not an issue, or can be tolerated in the rare case of failure. More sophisticated load balancing techniques generally keep track of what flows go to what servers, minimizing the disruption if one fails. They can also have better awareness of the load on each backend, and use more granular criteria to decide what flows go to which box. But if you can get away without that Anycasting works very well, and leaves all the heavy lifting to the switch ASIC.

Existing servers on production public subnets

Finally we should consider things like cloudcontrol, cloudservices, cloudstore and cloudelastic, which are on the current production public Vlan. Ideally whichever option is selected would also support migration of those to the cloud realm at some stage in the future.

I did not forget this task, but have been busy the last few days and was unable to take time and write down my thoughts, yet.

create and shared a spreadsheet trying to capture/compare the different options being considered.

We had a meeting today, rough summary:

The idea is roughly:

• get a couple new hardware servers (cloudlb?) dual homed (cloud-dedicated VLAN <-> cloud-host vlan, exact details TBD).
• give them a public IPv4 address. Only 1, for a VIP. This IP is allocated from a cloud-dedicated IPv4 pool/CIDR and it will be associated with the wikimediacloud.org domain.
• introduce keepalived (VRRP) and haproxy (proxying/loadbalancing) into the new servers
• have all new services be backends of the above. Initially cloudswift, others likely to follow (cloudcontrol, cloudservices, etc).

Our next steps will be:

• create a draft with the plans, diagrams and some initial implementation details -- Arturo to bootstrap this in the next few weeks.
• iterate over the draft until we feel it sounds like an actual plan --- both WMCS & SRE/IF to iterate over it.
• once we have a clear picture of what the architecture is and how the service will work, talk to the Traffic SRE team and coordinate with them. --- both WMCS & SRE/IF to participate in this meeting

Related to T314847