Page MenuHomePhabricator

lvs2011: Move existing row C & D vlans to primary uplink and add new ones
Closed, ResolvedPublic

Description

Now that the codfw row C and D vlans are connected to the EVPN switches in those rows we can use VXLAN to extend them to the top-of-rack switch in rack A2, and trunk them to the LVS there over its primary uplink.

This simplifies the setup and means we can remove the cross-rack links from the host to asw-c7-codfw and asw-d7-codfw.

Prep

In advance the current row C & D vlans (2003-2004, 2019-2020) need to be added to the list of tagged/trunked vlans on lsw1-a2-codfw xe-0/0/42, as well as the new ones (2036-2050). This can be done any time.

Window

On the day of the move it should be fairly striaghtforward:

  1. Downtime lvs2011 and lsw1-a2-codfw
  2. Disable BGP session to lvs2011 from lsw1-a2-codfw - this moves live traffic to lvs2014
  3. Merge puppet patch to reconfigure vlan setup on lvs2011
  4. Run puppet on lvs2011 to update /etc/network/interfaces
  5. (Optional) Reboot lvs2011
  6. Test connectivity to the new/moved vlans (presence of IPv6 autoconfigured IP should be sufficient)
  7. Re-enable puppet & PyBal on lvs2011

Tidy Up

  • Run PuppetDB import script for lvs2011 in Netbox
  • Remove cross-rack fibers
  • Default config for asw-c7-codfw xe-7/0/45
  • Default config for asw-d7-codfw xe-7/0/45

Event Timeline

cmooney triaged this task as Medium priority.

Change #1056521 had a related patch set uploaded (by Cathal Mooney; author: Cathal Mooney):

[operations/puppet@production] lvs2011: move row C & D vlans to primary uplink and add new ones

https://gerrit.wikimedia.org/r/1056521

Icinga downtime and Alertmanager silence (ID=fdb9ae19-db19-42c1-a837-d30eff237a37) set by cmooney@cumin1002 for 1:00:00 on 1 host(s) and their services with reason: reconfigure vlans on lvs2011

lsw1-a2-codfw.mgmt

Icinga downtime and Alertmanager silence (ID=0cfea209-8c6a-4d44-8fbf-96f5cd79d587) set by cmooney@cumin1002 for 1:00:00 on 1 host(s) and their services with reason: reconfigure vlans on lvs2011

lvs2011.codfw.wmnet

Mentioned in SAL (#wikimedia-operations) [2024-07-30T19:29:45Z] <topranks> disable BGP to lvs2011 on lsw1-a2-codfw (moves traffic to lvs2014) in advnace of vlan change T370891

Change #1056521 merged by Cathal Mooney:

[operations/puppet@production] lvs2011: move row C & D vlans to primary uplink and add new ones

https://gerrit.wikimedia.org/r/1056521

Mentioned in SAL (#wikimedia-operations) [2024-07-30T19:58:40Z] <topranks> rebooting lvs2011 to force new network config T370891

Mentioned in SAL (#wikimedia-operations) [2024-07-30T20:06:14Z] <topranks> re-enable BGP to lvs2011 on lsw1-a2-codfw (restores as primary for traffic) T370891

cmooney added a subscriber: Papaul.

Work completed on this one on the network & LVS side.

@Papaul we can now remove the cross-rack links any time we want, they're no longer in use.

https://netbox.wikimedia.org/dcim/interfaces/16695/trace/

https://netbox.wikimedia.org/dcim/interfaces/16696/trace/