Page MenuHomePhabricator

lvs2014: move uplink to lsw1-d2-codfw and connect to per-rack vlan
Closed, ResolvedPublic

Description

lvs2014 needs to be connected to one of our EVPN switches to enable connectivity to the new per-rack vlans configured on them in codfw rows C and D. To achieve this we will move it's primary uplink from asw-d2-codfw to lsw1-d2-codfw, and trunk the new vlans over the new port.

Additionally when doing the move we also plan to make some changes as described in the parent task:

  • Change the primary IP of the host, moving from the private1-d-codfw vlan to private1-d2-codfw
  • Change the BGP peering for the host, to peer with the top-of-rack switch (lsw1-d2-codfw) rather than core routers
    • These two changes mean traffic for internal services behind the LVS no longer need to traverse the core routers
  • Move the vlan sub-interfaces for the row A and B vlans to the host's newly-moved primary uplink
    • These are currently on a separate physical port that is connected cross-row to ssw1-a8-codfw

The details steps we need to carry out are below.

Prep

Before the change netops will configure the new switchport in Netbox and push the config to the switch so it is ready.

Switch:        lsw1-d2-codfw
Port:          xe-0/0/43
Untagged vlan: private1-d2-codfw
Tagged vlans:  private1-a*codfw, private1-b*codfw, priavte1-c*codfw, private1-d*codfw, public1-a*codfw, public1-b*codfw, public1-c*-codfw, public1-d*codfw

Window

  1. Downtime lvs2014 and lsw1-d2-codfw
  2. Set the 'bgp' flag for lvs2014 to false in Netbox
  3. Run homer against cr1-codfw and cr2-codfw to remove the BGP peering to lvs2014
  4. Manually update lvs2014 interface eno12399np0 in Netbox so it terminates on lsw1-d2-codfw xe-0/0/43
  5. Remove IPs 10.192.48.213/22 and 2620:0:860:104:10:192:48:213/64 from lvs2014 interface eno12399np0 in Netbox
  6. Remove DNS names for 10.192.48.213/22 and 2620:0:860:104:10:192:48:213/64 in Netbox
  7. Attach 10.192.37.5/24 and 2620:0:860:11c:10:192:37:5/64 to lvs2014 interface eno12399np0 in Netbox
  8. Make above two IPs 'primary' for lvs2014 in Netbox, and give them dns_name lvs2014.codfw.wmnet.
  9. Run the sre.dns.netbox cookbook to update the DNS entries for lvs2014
  10. Run the sre.dns.wipe-cache cookbook for lvs2014.codfw.wmnet
  11. Move the lvs2014 eno12399np0 physical link from asw-d2-codfw xe-2/0/44 to lsw1-d2-codfw xe-0/0/43
  12. Merge the puppet patch to re-define the vlan interface setup on lvs2014
  13. Run the sre.hosts.reimage cookbook for lvs2014
  14. After successful re-image check the networking looks good for primary and all vlan interfaces on lvs2014
  15. Change the BGP flag for lvs2014 in Netbox to 'true'
  16. Run Homer against lsw1-d2-codfw to add the BGP peering from switch to lvs2014.

Tidy Up

  • Run PuppetDB import script for lvs2014 in Netbox
  • Remove cross-rack fiber
  • Default config for ssw1-a8-codfw xe-0/0/32

Event Timeline

cmooney triaged this task as Medium priority.
cmooney renamed this task from Move lvs2014 uplink to lsw1-d2-codfw, connect to private1-d2-codfw and trunk all vlans on primary. to Move lvs2014 uplink to lsw1-d2-codfw and connect to per-rack vlan.Jul 24 2024, 3:39 PM
cmooney renamed this task from Move lvs2014 uplink to lsw1-d2-codfw and connect to per-rack vlan to lvs2014: move uplink to lsw1-d2-codfw and connect to per-rack vlan.

Change #1056550 had a related patch set uploaded (by Cathal Mooney; author: Cathal Mooney):

[operations/puppet@production] lvs2014: move A and B vlans to primary link and add new C and D vlans

https://gerrit.wikimedia.org/r/1056550

cmooney added a project: ops-codfw.
cmooney added a subscriber: Papaul.

Mentioned in SAL (#wikimedia-operations) [2024-08-20T16:22:47Z] <topranks> begginng work to reimage lvs2014 onto per-rack vlan in codfw rack D2 and move to new switch T370897

Icinga downtime and Alertmanager silence (ID=24f68f00-c864-474e-a3e6-c044aab86afa) set by cmooney@cumin1002 for 3:00:00 on 1 host(s) and their services with reason: move lvs2014 from asw to lsw

lsw1-d2-codfw.mgmt

Icinga downtime and Alertmanager silence (ID=612388e5-b8df-408f-81be-6f237cee6e7c) set by cmooney@cumin1002 for 3:00:00 on 1 host(s) and their services with reason: move lvs2014 from asw to lsw

lvs2014.codfw.wmnet

Mentioned in SAL (#wikimedia-operations) [2024-08-20T16:26:36Z] <topranks> disabling BGP to PyBal on lvs2014 in preparation for move to new switch T370897

Mentioned in SAL (#wikimedia-operations) [2024-08-20T16:40:44Z] <topranks> adding vlans to lsw1-d2-codfw for lvs2014 T370897

Change #1056550 merged by Cathal Mooney:

[operations/puppet@production] lvs2014: move A and B vlans to primary link and add new C and D vlans

https://gerrit.wikimedia.org/r/1056550

Cookbook cookbooks.sre.hosts.reimage was started by cmooney@cumin1002 for host lvs2014.codfw.wmnet with OS bullseye

Mentioned in SAL (#wikimedia-operations) [2024-08-20T17:16:28Z] <topranks> removing config for ssw1-a8-codfw link to lvs2014 T370897

Cookbook cookbooks.sre.hosts.reimage started by cmooney@cumin1002 for host lvs2014.codfw.wmnet with OS bullseye completed:

  • lvs2014 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Unable to disable Puppet, the host may have been unreachable
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202408201746_cmooney_3153929_lvs2014.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB

Icinga downtime and Alertmanager silence (ID=f69725b2-8f49-49fe-8766-ce7bb9ffa253) set by cmooney@cumin1002 for 2:00:00 on 2 host(s) and their services with reason: test failover lvs2013 to ls2014

cr[1-2]-codfw

Icinga downtime and Alertmanager silence (ID=b60d0e72-74ce-4dee-9bed-2acca82f8655) set by cmooney@cumin1002 for 2:00:00 on 1 host(s) and their services with reason: test failover lvs2013 to ls2014

lvs2013.codfw.wmnet

Work completed, no issues to report (although I had to downgrade the NIC firmware to 21.60.22.11 for PXEboot to start with the BCM57414 25G NIC and a 10G DAC, which we've seen before).

@Papaul we can remove these two additional cables, which are no longer in use. Both are yellow fibre links:

ssw1-a8-codfw xe-0/0/32 -->  lvs2014 eno12409np1
asw-c4-codfw  xe-4/0/47 -->  lvs2014 enp152s0f1np1

I mistakenly removed them from Netbox already, probably should have waited until they were physically removed first.

Task can be closed once they are tidied up. Thanks!

Papaul claimed this task.

Cookbook cookbooks.sre.hosts.reimage was started by cmooney@cumin1002 for host lvs2013.codfw.wmnet with OS bullseye

Cookbook cookbooks.sre.hosts.reimage started by cmooney@cumin1002 for host lvs2013.codfw.wmnet with OS bullseye completed:

  • lvs2013 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Unable to disable Puppet, the host may have been unreachable
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202408221611_cmooney_3608636_lvs2013.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB

Cookbook cookbooks.sre.hosts.reimage was started by cmooney@cumin1002 for host sretest2002.codfw.wmnet with OS bookworm

Cookbook cookbooks.sre.hosts.reimage started by cmooney@cumin1002 for host sretest2002.codfw.wmnet with OS bookworm executed with errors:

  • sretest2002 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • The reimage failed, see the cookbook logs for the details,You can also try typing "install-console" sretest2002.codfw.wmnet to get a root shellbut depending on the failure this may not work.

Cookbook cookbooks.sre.hosts.reimage was started by cmooney@cumin1002 for host sretest2002.codfw.wmnet with OS bookworm

Cookbook cookbooks.sre.hosts.reimage started by cmooney@cumin1002 for host sretest2002.codfw.wmnet with OS bookworm executed with errors:

  • sretest2002 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • The reimage failed, see the cookbook logs for the details,You can also try typing "install-console" sretest2002.codfw.wmnet to get a root shellbut depending on the failure this may not work.

Cookbook cookbooks.sre.hosts.reimage was started by cmooney@cumin1002 for host sretest2002.codfw.wmnet with OS bookworm

Cookbook cookbooks.sre.hosts.reimage started by cmooney@cumin1002 for host sretest2002.codfw.wmnet with OS bookworm executed with errors:

  • sretest2002 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • The reimage failed, see the cookbook logs for the details,You can also try typing "install-console" sretest2002.codfw.wmnet to get a root shellbut depending on the failure this may not work.