Page MenuHomePhabricator

lvs1020: reimage to move primary IP from private1-d-eqiad to private1-d7-eqiad vlan
Open, MediumPublic

Description

Once T405609 is complete we can look at this task.

Effectively there are two objectives:

  1. Reimage the host, moving it from the legacy row-wide vlan to the per-rack one available on the new switch
  2. Adjust the BGP config to peer with the switch directly, instead of with the CR routers

The real benefit of this is that the switches have a specific route for all the VIPs it announces, and thus traffic coming from our other hosts to services behind the load-balancer will take an optimal path through the network, avoiding the bottleneck of our core routers if it originates within row C or D.

Basic steps should be:

  • Create puppet patch so the LVS peers with the .1 address on the new vlan (the switch itself)
    • It should also create a new vlan sub-interface for the private1-d-eqiad vlan
  • Kick off a reimage of the host with the --move-vlan option set
  • Once the cookbook has edited the Netbox config, manually check all required vlans are still on the primary port
    • If not add them back in, --move-vlan probably is not designed to deal with those
    • Also add private1-c-eqiad to the list of tagged vlans on the switch port
  • Merge the puppet patch while the OS is installing
  • Run Homer against cr1-eqiad, cr2-eqiad and lsw1-d7-eqiad
    • This removes the old CR BGP conf, and adds the required BGP config and additional vlan trunk to the switch
  • When it comes back verify things look ok

Event Timeline

cmooney triaged this task as Medium priority.
cmooney updated the task description. (Show Details)

I'm actually not sure if this is going to be a possibility.

Unfortunately the Nokia SR-Linux platform does not allow us to set a BGP policy which matches routes based on the MED value on them. This is what we do on the Juniper switches to add the needed BGP community strings to flag the routes as prefered/backup etc.

set policy-options policy-statement LVS_import term add_avoid_community from protocol bgp
set policy-options policy-statement LVS_import term add_avoid_community from metric 200
set policy-options policy-statement LVS_import term add_avoid_community then community add AVOIDED_PATH
set policy-options policy-statement LVS_import term add_backup_community from protocol bgp
set policy-options policy-statement LVS_import term add_backup_community from metric 100
set policy-options policy-statement LVS_import term add_backup_community then community add BACKUP_PATH

Without this the MED set by PyBal will influence preference between different LVS hosts connected to the same IBGP cluster (in this case the Nokia switches in eqiad), but won't be visible to BGP speakers outside that (i.e. the core routers or Juniper switches in eqiad).

I can't really think of any way around this for now, we probably need to leave the LVS peering with the core routers, so that the community will be set. The downside there is all traffic from internal hosts needs to go up through the spine to core router and back down to the given LVS. Those links should have plenty of bandwidth though so could be worse.

Just to update this we are not planning to move the IP gateways for vlans to the Nokia switches in the short term (see T409286), and we won't be tackling this until that is done. So we can review the situation at that point and see where we are with Liberica.

@cmooney: We plan to move to Liberica in Q1 or Q2 of FY2026. Do you think we should still consider working on this?

@cmooney: We plan to move to Liberica in Q1 or Q2 of FY2026. Do you think we should still consider working on this?

Yes it absolutely has to be done. But we hit some blockers with moving the IP gateway to the Nokias (T420706) which means it doesn't make a huge difference for right now. There are also some advantages to keeping things as they are while we are still running PyBal as it's complicated to support the backup routes with it. So the sooner we move to Liberica the better.

Once we are looking at the gateway move post switch upgrade I'll be focusing on this again anyway whether we're upgraded or not (we can do some annoying manual policy stuff to support PyBal even on Nokia if it comes to it).

Thanks for the update and the explanation, @cmooney!