Page MenuHomePhabricator
Create Task
Maniphest T301142

Migrate lvs101[345] to lvs101[789]
Closed, ResolvedPublic
Actions

Description

The replacement LVSes were racked in T295804. This task is about going through the migration process to replace each of the old LVSes with a new one, which includes moving physical cables over for the non-primary interfaces.

lvs1016 was already migrated to lvs1020 before the holidays (its patches are in the earlier-linked task). We'll do the other three under this ticket.

  • lvs1013 -> lvs1017 (high-traffic1)
  • lvs1014 -> lvs1018 (high-traffic2)
  • lvs1015 -> lvs1019 (low-traffic)
  • lvs1016 -> lvs1020 (secondary; done in other task)

Details

Show related patches Customize query in gerrit

Related Objects

Event Timeline

BBlack created this task.Feb 7 2022, 4:03 PM
BBlack updated the task description. (Show Details)
cmooney subscribed.Feb 7 2022, 4:45 PM
Maintenance_bot added a project: SRE.Feb 7 2022, 4:45 PM
wiki_willy assigned this task to • Cmjohnson.Feb 7 2022, 4:58 PM
Vgutierrez subscribed.Feb 7 2022, 5:12 PM
gerritbot added a comment.Feb 10 2022, 6:39 PM

Change 761697 had a related patch set uploaded (by BBlack; author: BBlack):

[operations/puppet@production] lvs1017 interface/role setup

https://gerrit.wikimedia.org/r/761697

gerritbot added a project: Patch-For-Review.Feb 10 2022, 6:39 PM
Stashbot added a comment.Feb 10 2022, 6:43 PM

Mentioned in SAL (#wikimedia-operations) [2022-02-10T18:43:53Z] <bblack> lvs1013 - stopping puppet+pybal for move to lvs1017, high-traffic1 traffic fails over to lvs1020 for now - T301142

gerritbot added a comment.Feb 10 2022, 6:54 PM

Change 761697 merged by BBlack:

[operations/puppet@production] lvs1017 interface/role setup

https://gerrit.wikimedia.org/r/761697

Maintenance_bot removed a project: Patch-For-Review.Feb 10 2022, 7:10 PM
Stashbot added a comment.Feb 10 2022, 7:11 PM

Mentioned in SAL (#wikimedia-operations) [2022-02-10T19:11:04Z] <bblack> lvs1017 rebooting for sanity-check after prod config - T301142

gerritbot added a comment.Feb 10 2022, 7:18 PM

Change 761704 had a related patch set uploaded (by BBlack; author: BBlack):

[operations/puppet@production] lvs1017: fix public1-b/c interface ordering

https://gerrit.wikimedia.org/r/761704

gerritbot added a project: Patch-For-Review.Feb 10 2022, 7:18 PM

Change 761704 merged by BBlack:

[operations/puppet@production] lvs1017: fix public1-b/c interface ordering

https://gerrit.wikimedia.org/r/761704

Stashbot added a comment.Feb 10 2022, 7:25 PM

Mentioned in SAL (#wikimedia-operations) [2022-02-10T19:25:37Z] <bblack> lvs1017 reboot again for clean network config - T301142

gerritbot added a comment.Feb 10 2022, 8:04 PM

Change 761709 had a related patch set uploaded (by BBlack; author: BBlack):

[operations/homer/public@master] Add lvs1017 to pybal neighbors

https://gerrit.wikimedia.org/r/761709

gerritbot added a comment.Feb 10 2022, 8:06 PM

Change 761709 merged by BBlack:

[operations/homer/public@master] Add lvs1017 to pybal neighbors

https://gerrit.wikimedia.org/r/761709

Maintenance_bot removed a project: Patch-For-Review.Feb 10 2022, 8:10 PM
gerritbot added a comment.Feb 10 2022, 9:23 PM

Change 761725 had a related patch set uploaded (by BBlack; author: BBlack):

[operations/puppet@production] lvs1016: clean up unused hieradata

https://gerrit.wikimedia.org/r/761725

gerritbot added a project: Patch-For-Review.Feb 10 2022, 9:23 PM

Change 761726 had a related patch set uploaded (by BBlack; author: BBlack):

[operations/puppet@production] lvs1013: deconfigure towards spare::system

https://gerrit.wikimedia.org/r/761726

gerritbot added a comment.Feb 10 2022, 9:27 PM

Change 761725 merged by BBlack:

[operations/puppet@production] lvs1016: clean up unused hieradata

https://gerrit.wikimedia.org/r/761725

gerritbot added a comment.Feb 10 2022, 9:27 PM

Change 761726 merged by BBlack:

[operations/puppet@production] lvs1013: deconfigure towards spare::system

https://gerrit.wikimedia.org/r/761726

gerritbot added a comment.Feb 10 2022, 9:41 PM

Change 761728 had a related patch set uploaded (by BBlack; author: BBlack):

[operations/homer/public@master] Remove lvs1013 from pybal neighbors

https://gerrit.wikimedia.org/r/761728

gerritbot added a comment.Feb 10 2022, 9:53 PM

Change 761728 merged by BBlack:

[operations/homer/public@master] Remove lvs1013 from pybal neighbors

https://gerrit.wikimedia.org/r/761728

ops-monitoring-bot added a comment.Feb 10 2022, 10:04 PM

Cookbook cookbooks.sre.hosts.reimage was started by bblack@cumin1001 for host lvs1013.eqiad.wmnet with OS buster

BBlack updated the task description. (Show Details)Feb 10 2022, 10:06 PM
Maintenance_bot removed a project: Patch-For-Review.Feb 10 2022, 10:10 PM
ops-monitoring-bot added a comment.Feb 10 2022, 10:27 PM

Cookbook cookbooks.sre.hosts.reimage started by bblack@cumin1001 for host lvs1013.eqiad.wmnet with OS buster completed:

  • lvs1013 (PASS)
    • Downtimed on Icinga
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh buster OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202202102203_bblack_353_lvs1013.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
BBlack mentioned this in T300820: Inbound interface errors.Feb 10 2022, 10:32 PM
BBlack added a comment.Feb 10 2022, 10:37 PM

lvs1013 -> lvs1017 is complete, including cleanup.

Since the process is tricky to get right and is a corner case for so much of our automation, I've documented it loosely in etherpad for now, to help with the next two: https://etherpad.wikimedia.org/p/LVS-Migration

(note that LVS replacements are rare, but they do happen. We might want to clean this up further on the next two runs, and then put it in Wikitech for whichever poor soul has to do the next one, e.g. codfw hw refresh).

• Cmjohnson moved this task from Backlog to High Priority Task on the ops-eqiad board.Feb 14 2022, 5:35 PM
gerritbot added a comment.Feb 15 2022, 4:18 PM

Change 762852 had a related patch set uploaded (by BBlack; author: BBlack):

[operations/puppet@production] lvs1018 interface/role setup

https://gerrit.wikimedia.org/r/762852

gerritbot added a project: Patch-For-Review.Feb 15 2022, 4:18 PM
Stashbot added a comment.Feb 15 2022, 4:26 PM

Mentioned in SAL (#wikimedia-operations) [2022-02-15T16:26:07Z] <bblack> lvs1014 - downtimed - stopping puppet+pybal to fail traffic over to lvs1020 - T301142

gerritbot added a comment.Feb 15 2022, 4:37 PM

Change 762852 merged by BBlack:

[operations/puppet@production] lvs1018 interface/role setup

https://gerrit.wikimedia.org/r/762852

Stashbot added a comment.Feb 15 2022, 5:08 PM

Mentioned in SAL (#wikimedia-operations) [2022-02-15T17:08:53Z] <bblack> cr[12]-eqiad: manual edit static fallback route for high-traffic2 from lvs1014 to lvs1018 - T301142

Maintenance_bot removed a project: Patch-For-Review.Feb 15 2022, 5:10 PM
gerritbot added a comment.Feb 15 2022, 5:20 PM

Change 762874 had a related patch set uploaded (by BBlack; author: BBlack):

[operations/homer/public@master] Remove lvs1014 from pybal neighbors

https://gerrit.wikimedia.org/r/762874

gerritbot added a project: Patch-For-Review.Feb 15 2022, 5:20 PM
gerritbot added a comment.Feb 15 2022, 5:23 PM

Change 762876 had a related patch set uploaded (by BBlack; author: BBlack):

[operations/puppet@production] lvs1014: unconfigure towards spare::system

https://gerrit.wikimedia.org/r/762876

gerritbot added a comment.Feb 15 2022, 5:33 PM

Change 762874 merged by BBlack:

[operations/homer/public@master] Remove lvs1014 from pybal neighbors

https://gerrit.wikimedia.org/r/762874

gerritbot added a comment.Feb 15 2022, 5:36 PM

Change 762876 merged by BBlack:

[operations/puppet@production] lvs1014: unconfigure towards spare::system

https://gerrit.wikimedia.org/r/762876

ops-monitoring-bot added a comment.Feb 15 2022, 5:48 PM

Cookbook cookbooks.sre.hosts.reimage was started by bblack@cumin1001 for host lvs1014.eqiad.wmnet with OS buster

gerritbot added a comment.Feb 15 2022, 6:02 PM

Change 762888 had a related patch set uploaded (by BBlack; author: BBlack):

[operations/puppet@production] lvs1019 interface/role setup

https://gerrit.wikimedia.org/r/762888

Stashbot added a comment.Feb 15 2022, 6:05 PM

Mentioned in SAL (#wikimedia-operations) [2022-02-15T18:05:41Z] <bblack> lvs1015 - stopping puppet+pybal to begin transition to lvs1019 - T301142

gerritbot added a comment.Feb 15 2022, 6:11 PM

Change 762888 merged by BBlack:

[operations/puppet@production] lvs1019 interface/role setup

https://gerrit.wikimedia.org/r/762888

ops-monitoring-bot added a comment.Feb 15 2022, 6:12 PM

Cookbook cookbooks.sre.hosts.reimage started by bblack@cumin1001 for host lvs1014.eqiad.wmnet with OS buster completed:

  • lvs1014 (PASS)
    • Downtimed on Icinga
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh buster OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202202151748_bblack_5466_lvs1014.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
Stashbot added a comment.Feb 15 2022, 6:41 PM

Mentioned in SAL (#wikimedia-operations) [2022-02-15T18:41:52Z] <bblack> lvs1019 - disable puppet/pybal, reboot - T301142

gerritbot added a comment.Feb 15 2022, 6:47 PM

Change 762903 had a related patch set uploaded (by BBlack; author: BBlack):

[operations/homer/public@master] Add lvs1019 to pybal neighbors

https://gerrit.wikimedia.org/r/762903

gerritbot added a comment.Feb 15 2022, 6:51 PM

Change 762903 merged by BBlack:

[operations/homer/public@master] Add lvs1019 to pybal neighbors

https://gerrit.wikimedia.org/r/762903

gerritbot added a comment.Feb 15 2022, 7:13 PM

Change 762930 had a related patch set uploaded (by BBlack; author: BBlack):

[operations/homer/public@master] Remove lvs1015 from pybal neighbors

https://gerrit.wikimedia.org/r/762930

gerritbot added a comment.Feb 15 2022, 7:15 PM

Change 762931 had a related patch set uploaded (by BBlack; author: BBlack):

[operations/puppet@production] lvs1015: unconfigure towards spare::system

https://gerrit.wikimedia.org/r/762931

gerritbot added a comment.Feb 15 2022, 7:16 PM

Change 762930 merged by BBlack:

[operations/homer/public@master] Remove lvs1015 from pybal neighbors

https://gerrit.wikimedia.org/r/762930

gerritbot added a comment.Feb 15 2022, 7:19 PM

Change 762931 merged by BBlack:

[operations/puppet@production] lvs1015: unconfigure towards spare::system

https://gerrit.wikimedia.org/r/762931

ops-monitoring-bot added a comment.Feb 15 2022, 7:30 PM

Cookbook cookbooks.sre.hosts.reimage was started by bblack@cumin1001 for host lvs1015.eqiad.wmnet with OS buster

BBlack updated the task description. (Show Details)Feb 15 2022, 7:31 PM
ops-monitoring-bot added a comment.Feb 15 2022, 7:51 PM

Cookbook cookbooks.sre.hosts.reimage started by bblack@cumin1001 for host lvs1015.eqiad.wmnet with OS buster completed:

  • lvs1015 (PASS)
    • Downtimed on Icinga
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh buster OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202202151929_bblack_4611_lvs1015.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
BBlack closed this task as Resolved.Feb 15 2022, 7:54 PM
gerritbot added a comment.Feb 15 2022, 8:03 PM

Change 762941 had a related patch set uploaded (by BBlack; author: BBlack):

[operations/puppet@production] Fix section label in smokeping config

https://gerrit.wikimedia.org/r/762941

gerritbot added a comment.Feb 15 2022, 8:04 PM

Change 762941 merged by BBlack:

[operations/puppet@production] Fix section label in smokeping config

https://gerrit.wikimedia.org/r/762941

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits