Page MenuHomePhabricator
Create Task
Maniphest T307997

Upgrade ganeti/ulsfo to Bullseye
Closed, ResolvedPublic
Actions

Description

Upgrade the Ganeti cluster in ulsfo to Bullseye:

  • Add component/ganeti3 to all nodes
  • Upgrade all nodes to the new Ganeti 3
  • sudo gnt-cluster upgrade --to 3.0

Empty every node of primary/secondary instances. Remove it with gnt-node remove and reimage it to Bullseye, then re-add with gnt-node add:

  • ganeti4001
  • ganeti4002
  • ganeti4003

Details

SubjectRepoBranchLines +/-
Enable ganeti3 component in ulsfooperations/puppetproduction+1 -3
Customize query in gerrit

Related Objects

Event Timeline

• MoritzMuehlenhoff created this task.May 10 2022, 10:39 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 10 2022, 10:39 AM
• MoritzMuehlenhoff triaged this task as Medium priority.May 10 2022, 10:39 AM
gerritbot added a comment.May 10 2022, 10:52 AM

Change 790643 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Enable ganeti3 component in ulsfo

https://gerrit.wikimedia.org/r/790643

gerritbot added a project: Patch-For-Review.May 10 2022, 10:52 AM
gerritbot added a comment.May 10 2022, 12:15 PM

Change 790643 merged by Muehlenhoff:

[operations/puppet@production] Enable ganeti3 component in ulsfo

https://gerrit.wikimedia.org/r/790643

• MoritzMuehlenhoff updated the task description. (Show Details)May 10 2022, 12:18 PM
Maintenance_bot removed a project: Patch-For-Review.May 10 2022, 12:30 PM
Stashbot added a comment.May 11 2022, 7:05 AM

Mentioned in SAL (#wikimedia-operations) [2022-05-11T07:05:18Z] <moritzm> updating ganeti4* to Ganeti 3.0.1-1~bpo10+1 T307997

Stashbot added a comment.May 11 2022, 7:18 AM

Mentioned in SAL (#wikimedia-operations) [2022-05-11T07:18:21Z] <moritzm> drain ganeti4001 T307997

ops-monitoring-bot added a comment.May 11 2022, 9:57 AM

Cookbook cookbooks.sre.hosts.reimage was started by jmm@cumin1001 for host ganeti4001.ulsfo.wmnet with OS bullseye

• MoritzMuehlenhoff updated the task description. (Show Details)May 11 2022, 9:58 AM
ops-monitoring-bot added a comment.May 11 2022, 10:25 AM

Cookbook cookbooks.sre.hosts.reimage started by jmm@cumin1001 for host ganeti4001.ulsfo.wmnet with OS bullseye executed with errors:

  • ganeti4001 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.May 11 2022, 12:45 PM

Cookbook cookbooks.sre.hosts.reimage was started by jmm@cumin1001 for host ganeti4001.ulsfo.wmnet with OS bullseye

ops-monitoring-bot added a comment.May 11 2022, 1:13 PM

Cookbook cookbooks.sre.hosts.reimage started by jmm@cumin1001 for host ganeti4001.ulsfo.wmnet with OS bullseye executed with errors:

  • ganeti4001 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.May 11 2022, 1:54 PM

Cookbook cookbooks.sre.hosts.reimage was started by jmm@cumin1001 for host ganeti4001.ulsfo.wmnet with OS bullseye

ops-monitoring-bot added a comment.May 11 2022, 2:22 PM

Cookbook cookbooks.sre.hosts.reimage started by jmm@cumin1001 for host ganeti4001.ulsfo.wmnet with OS bullseye executed with errors:

  • ganeti4001 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details
Stashbot added a comment.May 11 2022, 3:15 PM

Mentioned in SAL (#wikimedia-sre) [2022-05-11T15:15:51Z] <robh> ganeti4001 updating all firmware revisions T307997\

Stashbot added a comment.May 11 2022, 3:53 PM

Mentioned in SAL (#wikimedia-sre) [2022-05-11T15:53:04Z] <robh> firmware upgrade for ganeti4001 complete T307997 (bios, nics, idrac) and manually confirmed first 10G port is link active (it is) and is set to pxe

ops-monitoring-bot added a comment.May 12 2022, 7:16 AM

Cookbook cookbooks.sre.hosts.reimage was started by jmm@cumin1001 for host ganeti4001.ulsfo.wmnet with OS bullseye

ops-monitoring-bot added a comment.May 12 2022, 7:34 AM

Cookbook cookbooks.sre.hosts.reimage started by jmm@cumin1001 for host ganeti4001.ulsfo.wmnet with OS bullseye executed with errors:

  • ganeti4001 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details
• MoritzMuehlenhoff added a comment.May 12 2022, 7:45 AM

Without a firmware update of the system firmware and the NIC firmware, there was no network link in the Debian installer. Rob updated everything to the latest version, but it seems now we're running into this firmware regression that Papaul previously identified for cloudvirt1025/cloudvirt1026: https://phabricator.wikimedia.org/T304483 , the initial PXE boot fails with "Failed to load ldlinux.c32", so we'll need a similar downgrade here (both servers use the same NICs)

RobH subscribed.May 12 2022, 3:53 PM

I've downgraded ganeti4001 to 21.60.22.11, flashed and confirmed. @MoritzMuehlenhoff give it a shot now!

ops-monitoring-bot added a comment.May 12 2022, 5:08 PM

Cookbook cookbooks.sre.hosts.reimage was started by jmm@cumin1001 for host ganeti4001.ulsfo.wmnet with OS bullseye

RobH added a comment.May 12 2022, 5:24 PM

Ok, since the last update, it now shows no connection on the switch. However, previous firmware versions all successfully had a link light on the switch (I checked when I updated to the newest one on may 11). Currently troubleshooting.

ops-monitoring-bot added a comment.May 12 2022, 5:26 PM

Cookbook cookbooks.sre.hosts.reimage started by jmm@cumin1001 for host ganeti4001.ulsfo.wmnet with OS bullseye executed with errors:

  • ganeti4001 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.May 12 2022, 5:51 PM

Cookbook cookbooks.sre.hosts.reimage was started by robh@cumin1001 for host ganeti4001.ulsfo.wmnet with OS bullseye

ops-monitoring-bot added a comment.May 12 2022, 6:26 PM

Cookbook cookbooks.sre.hosts.reimage started by robh@cumin1001 for host ganeti4001.ulsfo.wmnet with OS bullseye completed:

  • ganeti4001 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202205121751_robh_1295746_ganeti4001.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
RobH added a comment.May 12 2022, 6:29 PM

Summary of work:

  • initially asked to update firmware, updated to the very latest 22.00.07.60
    • this introduced a new pxe boot failure known via T304483, Moritz requested I match to the working version 21.60.22.11 (works on cloudvirts in eqiad.)
  • flashed 10g nic firmware to 21.60.22.11
    • this introduced a new error of the link light/confirmation of connection going away for no explainable reason, as the settings were all consistent for a working NIC but the link was gone
  • flashed 10g nic firmware to 21.85.21.92, a release just under newest release 22.00.07.60
    • link light verification/port info in idrac interface returned, link is in effect
    • launched install via cookbook and while the tftp load of the debian installer seemed to take awhile (about 4 minutes), it did eventually load up and the installer loaded and ran successfully.
• MoritzMuehlenhoff updated the task description. (Show Details)May 13 2022, 7:53 AM
Stashbot added a comment.May 13 2022, 7:59 AM

Mentioned in SAL (#wikimedia-operations) [2022-05-13T07:59:10Z] <moritzm> draining ganeti4002 T307997

• MoritzMuehlenhoff added a comment.May 13 2022, 10:51 AM

ganeti4002 is from the same batch. I've migrated instances, removed it from the cluster for the reimage and downtimed it. @RobH Can you please update it to the same firmware and NIC firmware versions as ganeti4001?

• MoritzMuehlenhoff reassigned this task from • MoritzMuehlenhoff to RobH.May 13 2022, 10:52 AM
Stashbot added a comment.May 16 2022, 9:47 PM

Mentioned in SAL (#wikimedia-sre) [2022-05-16T21:47:17Z] <robh> ganeti4002 rebooting for firmware update via T307997

RobH added a comment.May 16 2022, 10:03 PM

Mortiz,

I flashed updates to ganeti4002, but it reminded me I need to go onsite this Thursday for T303318 to swap the defective memory (existing open case before the warranty expired)

ops-monitoring-bot added a comment.May 17 2022, 8:48 AM

Cookbook cookbooks.sre.hosts.reimage was started by jmm@cumin1001 for host ganeti4002.ulsfo.wmnet with OS bullseye

ops-monitoring-bot added a comment.May 17 2022, 9:25 AM

Cookbook cookbooks.sre.hosts.reimage started by jmm@cumin1001 for host ganeti4002.ulsfo.wmnet with OS bullseye completed:

  • ganeti4002 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202205170848_jmm_2136727_ganeti4002.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
Stashbot added a comment.May 17 2022, 11:53 AM

Mentioned in SAL (#wikimedia-operations) [2022-05-17T11:53:51Z] <moritzm> failover Ganeti master in ulsfo to ganeti4001 T307997

Stashbot added a comment.May 17 2022, 12:04 PM

Mentioned in SAL (#wikimedia-operations) [2022-05-17T12:04:16Z] <moritzm> draining ganeti4003 T307997

Volans mentioned this in T307538: Write a GitLab "Migrating a Project" runbook / manual.May 17 2022, 2:34 PM
• MoritzMuehlenhoff updated the task description. (Show Details)May 17 2022, 5:07 PM

ganeti4003 is from the same batch and needs the same updates. I've migrated instances, removed it from the cluster for the reimage and downtimed it.

Stashbot added a comment.May 17 2022, 5:16 PM

Mentioned in SAL (#wikimedia-sre) [2022-05-17T17:16:44Z] <robh> ganeti4003 rebooting for firmware updates via T307997

RobH added a comment.May 17 2022, 6:43 PM

updates all done, system is back up for reimage whenever

RobH reassigned this task from RobH to • MoritzMuehlenhoff.May 17 2022, 6:44 PM
ops-monitoring-bot added a comment.May 18 2022, 7:59 AM

Cookbook cookbooks.sre.hosts.reimage was started by jmm@cumin2002 for host ganeti4003.ulsfo.wmnet with OS bullseye

ops-monitoring-bot added a comment.May 18 2022, 8:33 AM

Cookbook cookbooks.sre.hosts.reimage started by jmm@cumin2002 for host ganeti4003.ulsfo.wmnet with OS bullseye completed:

  • ganeti4003 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202205180756_jmm_1188152_ganeti4003.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
RobH unsubscribed.May 18 2022, 5:58 PM
• MoritzMuehlenhoff updated the task description. (Show Details)Jun 17 2022, 10:24 AM
• MoritzMuehlenhoff closed this task as Resolved.Jun 17 2022, 10:27 AM

This is complete. The ulsfo cluster is affected by T309724, but that will be investigated via that task (and it doesn't have a functional impact apart from the fact that gnt-cluster verify fails)

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits