Page MenuHomePhabricator

lvs1019: reimage to move primary IP from private1-c-eqiad to private1-c7-eqiad vlan
Open, MediumPublic

Description

Once T405628 is complete we can look at this task.

Effectively there are two objectives:

  1. Reimage the host, moving it from the legacy row-wide vlan to the per-rack one available on the new switch
  2. Adjust the BGP config to peer with the switch directly, instead of with the CR routers

The real benefit of this is that the switches have a specific route for all the VIPs it announces, and thus traffic coming from our other hosts to services behind the load-balancer will take an optimal path through the network, avoiding the bottleneck of our core routers if it originates within row C or D.

Basic steps should be:

  • Create puppet patch so the LVS peers with the .1 address on the new vlan (the switch itself)
    • The puppet patch should also add config for private1-c-eqiad to be a new vlan sub-interface of the primary
  • Downtime the host
  • Set the 'bgp' flag to false in Netbox for the host
  • Run Homer against cr1-eqiad, cr2-eqiad to remove the BGP config for the host on the CRs
  • Observe live stats to verify that traffic re-directs to the backup lvs1020 without an issue, and connections go to zero
  • Kick off a reimage of the host with the --move-vlan option set
  • Once the cookbook has edited the Netbox config, manually check all required vlans are still on the primary port
    • If not add them back in, --move-vlan probably is not designed to deal with those
    • Also add private1-c-eqiad
  • Merge the puppet patch while the OS is installing
  • When it comes back verify the config looks ok and setup is good
  • Run Homer against lsw1-c7-eqiad to add the BGP config and trunk private1-c-eqiad on the port
  • Validate BGP is back up, routes announced
  • Check that connections start rising and it takes over from the backup successfully

Event Timeline

cmooney triaged this task as Medium priority.
cmooney renamed this task from lvs1020: reimage to move primary IP from private1-c-eqiad to private1-c7-eqiad vlan to lvs1019: reimage to move primary IP from private1-c-eqiad to private1-c7-eqiad vlan.Sep 25 2025, 5:36 PM
cmooney updated the task description. (Show Details)

See T405630#11250099, I'm not sure this will be possible.

Just to update this we are not planning to move the IP gateways for vlans to the Nokia switches in the short term (see T409286), and we won't be tackling this until that is done. So we can review the situation at that point and see where we are with Liberica.