Page MenuHomePhabricator

Eqiad row C/D switch refresh: LVS changes to support migration
Open, MediumPublic

Description

As with previous switch migrations from our legacy layer-2 based solution to the routed spine/leaf setup we need to take special account of our LVS load-balancers to make sure we are able to migrate everything smoothly.

The principal things we need to do are:

  1. Move the primary uplink of the LVS servers in rows C/D to the new switches prior to moving other hosts
  2. For low-traffic LVS trunk the new per-rack vlans we have in rows C/D to them directly over their primary link
    1. And add the puppet config to support
  3. Remove any redundant links (due to more vlans trunked on the primary port rather than a cross-rack link)
  4. Reimage any LVS hosts in the affected rows to move them to the pre-rack vlan instead of row-wide
    1. Adjusting the BGP configuration in the process to peer with the top-of-rack switch
    2. And ensuring traffic sent to the load-balancer flows entirely switch<->switch where possible, avoiding CR bottleneck

Luckily the scenario is simpler in this case for a few reasons:

  1. Only low-traffic services use direct L2 adjacency
  2. The high-traffic LVS, lvs1016 and lvs1018, are not in the rows to be migrated
  3. lvs1019 and lvs1020 are in rows C and D respectively, so we can remove some of their cross-rack links and simplify things

I will open sub-tasks for each of the changes we need to complete.

Event Timeline

cmooney triaged this task as Medium priority.