Page MenuHomePhabricator

lvs2013: move uplink to lsw1-c2-codfw and connect to per-rack vlan
Closed, ResolvedPublic

Description

lvs2013 needs to be connected to one of our EVPN switches to enable connectivity to the new per-rack vlans configured on them in codfw rows C and D. To achieve this we will move it's primary uplink from asw-c2-codfw to lsw1-c2-codfw, and trunk the new vlans over the new port.

Additionally when doing the move we also plan to make some changes as described in the parent task:

  • Change the primary IP of the host, moving from the private1-c-codfw vlan to private1-c2-codfw
  • Change the BGP peering for the host, to peer with the top-of-rack switch (lsw1-c2-codfw) rather than core routers
    • These two changes mean traffic for internal services behind the LVS no longer need to traverse the core routers
  • Move the vlan sub-interfaces for the row A and B vlans to the host's newly-moved primary uplink
    • These are currently on a separate physical port that is connected cross-row to ssw1-a1-codfw

The details steps we need to carry out are below.

Prep

Before the change netops will configure the new switchport in Netbox and push the config to the switch so it is ready.

Switch:        lsw1-c2-codfw
Port:          xe-0/0/43
Untagged vlan: private1-c2-codfw
Tagged vlans:  private1-a*codfw, private1-b*codfw, priavte1-c*codfw, private1-d*codfw, public1-a*codfw, public1-b*codfw, public1-c*-codfw, public1-d*codfw

Window

  1. Downtime lvs2013 and lsw1-c2-codfw
  2. Set the 'bgp' flag for lvs2013 to false in Netbox
  3. Run homer against cr1-codfw and cr2-codfw to remove the BGP peering to lvs2013
    • This will cause lvs2014 to take over for service IPs previously routed to lvs2013
    • We need to wait and check connections move over as expected before proceeding
  4. Manually update lvs2013 interface eno12399np0 in Netbox so it terminates on lsw1-c2-codfw xe-0/0/43
  5. Remove IPs 10.192.32.14/22 and 2620:0:860:103:10:192:32:14/64 from lvs2013 interface eno12399np0 in Netbox
  6. Remove DNS names for 10.192.32.14/22 and 2620:0:860:103:10:192:32:14/64 in Netbox
  7. Attach 10.192.26.4/24 and 2620:0:860:105:10:192:26:4/64 to lvs2013 interface eno12399np0 in Netbox
  8. Make above two IPs 'primary' for lvs2013 in Netbox, and give them dns_name lvs2013.codfw.wmnet.
  9. Run the sre.dns.netbox cookbook to update the DNS entries for lvs2013
  10. Run the sre.dns.wipe-cache cookbook for lvs2013.codfw.wmnet
  11. Move the lvs2013 eno12399np0 physical link from asw-c2-codfw xe-2/0/44 to lsw1-c2-codfw xe-0/0/43
  12. Merge the puppet patch to re-define the vlan interface setup on lvs2013
  13. Run the sre.hosts.reimage cookbook for lvs2013
  14. After successful re-image check the networking looks good for primary and all vlan interfaces on lvs2013
  15. Change the BGP flag for lvs2013 in Netbox to 'true'
  16. Run Homer against lsw1-c2-codfw to add the BGP peering from switch to lvs2013.

Tidy Up

  • Run PuppetDB import script for lvs2013 in Netbox
  • Remove cross-rack fiber
  • Default config for ssw1-a1-codfw xe-0/0/32

Event Timeline

Change #1056563 had a related patch set uploaded (by Cathal Mooney; author: Cathal Mooney):

[operations/puppet@production] lvs2013: move A and B vlans to primary link and add new C and D vlans

https://gerrit.wikimedia.org/r/1056563

cmooney updated the task description. (Show Details)
cmooney triaged this task as Medium priority.Jul 29 2024, 2:30 PM

Icinga downtime and Alertmanager silence (ID=4888a1d9-ee36-415c-a204-98c84040effe) set by cmooney@cumin1002 for 2:00:00 on 1 host(s) and their services with reason: move lvs2013 from asw to lsw

lsw1-c2-codfw.mgmt

Icinga downtime and Alertmanager silence (ID=028cbb12-db86-4824-9084-463287cc8911) set by cmooney@cumin1002 for 2:00:00 on 1 host(s) and their services with reason: move lvs2013 from asw to lsw

lvs2013.codfw.wmnet

Mentioned in SAL (#wikimedia-operations) [2024-08-22T15:31:09Z] <topranks> disabling BGP on cr1-codfw and cr2-codfw towards lvs2013 in advance of host move to new switch T370927

Mentioned in SAL (#wikimedia-operations) [2024-08-22T15:36:49Z] <topranks> add vlans to trunk port on lsw1-c2-codfw facing new lvs2013 link T370927

Change #1056563 merged by Cathal Mooney:

[operations/puppet@production] lvs2013: move A and B vlans to primary link and add new C and D vlans

https://gerrit.wikimedia.org/r/1056563

Mentioned in SAL (#wikimedia-operations) [2024-08-22T17:10:54Z] <topranks> removing no-longer-required vlans from ssw1-a1-codfw after lvs move T370927

cmooney added a subscriber: Jhancock.wm.

All work completed, no issues to report.

@Jhancock.wm @Papaul these two cross-rack fibre links can be removed at any stage:

https://netbox.wikimedia.org/dcim/interfaces/21086/trace/

https://netbox.wikimedia.org/dcim/interfaces/29759/trace/

And then cables deleted in Netbox and this task closed. Thanks!