Page MenuHomePhabricator

lvs2012: Move existing row C & D vlans to primary uplink and add new ones
Closed, ResolvedPublic

Description

Now that the codfw row C and D vlans are connected to the EVPN switches in those rows we can use VXLAN to extend them to the top-of-rack switch in rack B2, and trunk them to the LVS there over its primary uplink.

This simplifies the setup and means we can remove the cross-rack links from the host to asw-c2-codfw and asw-d4-codfw.

Prep

In advance the current row C & D vlans (2003-2004, 2019-2020) need to be added to the list of tagged/trunked vlans on lsw1-b2-codfw xe-0/0/42, as well as the new ones (2036-2050). This can be done any time.

Window

On the day of the move it should be fairly striaghtforward:

  1. Downtime lvs2012 and lsw1-b2-codfw
  2. Disable BGP session to lvs2012 from lsw1-b2-codfw - this moves live traffic to lvs2014
  3. Merge puppet patch to reconfigure vlan setup on lvs2012
  4. Run puppet on lvs2012 to update /etc/network/interfaces
  5. (Optional) Reboot lvs2012
  6. Test connectivity to the new/moved vlans (presence of IPv6 autoconfigured IP should be sufficient)
  7. Re-enable puppet & PyBal on lvs2012

Tidy Up

  • Run PuppetDB import script for lvs2012 in Netbox
  • Remove cross-rack fibers
  • Default config for asw-c2-codfw xe-2/0/45
  • Default config for asw-d4-codfw xe-4/0/47

Event Timeline

cmooney triaged this task as Medium priority.

Change #1056478 had a related patch set uploaded (by Cathal Mooney; author: Cathal Mooney):

[operations/puppet@production] lvs2012: move row C & D vlans to primary uplink and and new ones

https://gerrit.wikimedia.org/r/1056478

cmooney updated the task description. (Show Details)
cmooney updated the task description. (Show Details)

Mentioned in SAL (#wikimedia-operations) [2024-07-30T17:12:56Z] <topranks> adding row C/D vlans to lsw1-b2-codfw and adding on trunk to lvs2012 T370862

Icinga downtime and Alertmanager silence (ID=22b0edee-c7a6-4b0f-9fea-2095ec629cd6) set by cmooney@cumin1002 for 0:40:00 on 1 host(s) and their services with reason: reconfigure vlans on lvs2012

lvs2012.codfw.wmnet

Icinga downtime and Alertmanager silence (ID=6ff7dee3-4248-4c63-812a-befb7aa3679f) set by cmooney@cumin1002 for 0:40:00 on 1 host(s) and their services with reason: reconfigure vlans on lvs2012

lsw1-b2-codfw.mgmt

Change #1056478 merged by Cathal Mooney:

[operations/puppet@production] lvs2012: move row C & D vlans to primary uplink and add new ones

https://gerrit.wikimedia.org/r/1056478

Icinga downtime and Alertmanager silence (ID=dd309020-6739-44e3-aae7-1db7e069b053) set by cmooney@cumin1002 for 0:30:00 on 1 host(s) and their services with reason: reconfigure vlans on lvs2012

lsw1-b2-codfw.mgmt

Icinga downtime and Alertmanager silence (ID=e014b03e-5922-4caa-80c4-c950cc419ff0) set by cmooney@cumin1002 for 0:30:00 on 1 host(s) and their services with reason: reconfigure vlans on lvs2012

lvs2012.codfw.wmnet

Mentioned in SAL (#wikimedia-operations) [2024-07-30T17:56:24Z] <topranks> rebooting lvs2012 to force new network config T370862

Icinga downtime and Alertmanager silence (ID=a53d3f9e-80ae-429e-b814-01f035f87c23) set by cmooney@cumin1002 for 0:30:00 on 1 host(s) and their services with reason: reconfigure vlans on lvs2012

lvs2012.codfw.wmnet

Mentioned in SAL (#wikimedia-operations) [2024-07-30T18:27:40Z] <topranks> rebooting lvs2012 (again) to force new network config T370862

Icinga downtime and Alertmanager silence (ID=428f84f9-4ca7-4d64-ba2f-941c3927470b) set by cmooney@cumin1002 for 0:30:00 on 1 host(s) and their services with reason: reconfigure vlans on lvs2012

lsw1-b2-codfw.mgmt

Icinga downtime and Alertmanager silence (ID=fea7df87-a776-4ad1-b5ea-1c4c47a61055) set by cmooney@cumin1002 for 0:30:00 on 1 host(s) and their services with reason: reconfigure vlans on lvs2012

lvs2012.codfw.wmnet

Mentioned in SAL (#wikimedia-operations) [2024-07-30T18:39:20Z] <topranks> re-enabling BGP to lvs2012 from lsw1-b2-codfw T370862

cmooney added a subscriber: Papaul.

Work on this one is completed, all that remains is to remove the old cross-rack links which are no longer being used.

@Papaul these two cross-rack links can be removed any time:

https://netbox.wikimedia.org/dcim/interfaces/16699/trace/

https://netbox.wikimedia.org/dcim/interfaces/21085/trace/

thanks!

@cmooney links removed. You can resolve the task if nothing else needs to be done.

@cmooney links removed. You can resolve the task if nothing else needs to be done.

Thanks!