Page MenuHomePhabricator
Create Task
Maniphest T308238

Upgrade ganeti/esams to Bullseye
Closed, ResolvedPublic
Actions

Description

Upgrade the Ganeti cluster in Amsterdam to Bullseye:

  • Add component/ganeti3 to all nodes
  • Upgrade all nodes to the new Ganeti 3
  • sudo gnt-cluster upgrade --to 3.0

Empty every node of primary/secondary instances. Remove it with gnt-node remove and reimage it to Bullseye, then re-add.

  • ganeti3001
  • ganeti3002
  • ganeti3003

Details

ProjectBranchLines +/-Subject
operations/puppetproduction+1 -0Enable component/ganeti3 for the esams cluster
Customize query in gerrit

Event Timeline

MoritzMuehlenhoff created this task.May 12 2022, 12:26 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 12 2022, 12:26 PM
MoritzMuehlenhoff triaged this task as Medium priority.May 12 2022, 12:26 PM
gerritbot added a comment.May 19 2022, 2:52 PM

Change 793491 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Enable component/ganeti3 for the esams cluster

https://gerrit.wikimedia.org/r/793491

gerritbot added a project: Patch-For-Review.May 19 2022, 2:52 PM
gerritbot added a comment.May 20 2022, 6:19 AM

Change 793491 merged by Muehlenhoff:

[operations/puppet@production] Enable component/ganeti3 for the esams cluster

https://gerrit.wikimedia.org/r/793491

Maintenance_bot removed a project: Patch-For-Review.May 20 2022, 6:30 AM
MoritzMuehlenhoff updated the task description. (Show Details)May 20 2022, 7:47 AM
Stashbot added a comment.Jun 7 2022, 7:44 AM

Mentioned in SAL (#wikimedia-operations) [2022-06-07T07:44:52Z] <moritzm> upgrading ganeti/esams to Ganeti 3 T308238

MoritzMuehlenhoff updated the task description. (Show Details)Jun 7 2022, 7:49 AM
MoritzMuehlenhoff updated the task description. (Show Details)Jun 7 2022, 8:17 AM
Stashbot added a comment.Jun 7 2022, 8:29 AM

Mentioned in SAL (#wikimedia-operations) [2022-06-07T08:29:54Z] <moritzm> drain ganeti3003 for reimage T308238

MoritzMuehlenhoff reassigned this task from MoritzMuehlenhoff to RobH.Jun 7 2022, 10:07 AM

ganeti3003 is removed from the cluster, downtimed and needs the same firmware/NIC updates as ganeti4*/ganeti5* to enable the reimage to Bullseye.

Stashbot added a comment.Wed, Jun 8, 4:26 PM

Mentioned in SAL (#wikimedia-sre) [2022-06-08T16:26:37Z] <robh> ganeti5003 firmware updates in progress via T308238

Stashbot added a comment.Wed, Jun 8, 4:44 PM

Mentioned in SAL (#wikimedia-sre) [2022-06-08T16:44:48Z] <robh> ganeti3003 (already depooled) coming down for firmware update and reimage via T308238

RobH added a comment.Wed, Jun 8, 5:14 PM

ganeti3003 firmware updates

bios 2.2.11 to 2.14.2
nic 21.40.22.20 to 21.85.21.92
idrac 3.34.34.34 to 5.10.10.00

ops-monitoring-bot added a comment.Wed, Jun 8, 5:15 PM

Cookbook cookbooks.sre.hosts.reimage was started by robh@cumin1001 for host ganeti3003.esams.wmnet with OS bullseye

ops-monitoring-bot added a comment.Wed, Jun 8, 5:54 PM

Cookbook cookbooks.sre.hosts.reimage started by robh@cumin1001 for host ganeti3003.esams.wmnet with OS bullseye completed:

  • ganeti3003 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202206081715_robh_441138_ganeti3003.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
RobH reassigned this task from RobH to MoritzMuehlenhoff.Wed, Jun 8, 5:57 PM
RobH added a subscriber: RobH.

ganeti3003 firmware updated and reimaged to bullseye (easy enough to fire the cookbook to reimage post firmware update to ensure the firmware update fixes pxe issues)

Stashbot added a comment.Thu, Jun 9, 7:13 AM

Mentioned in SAL (#wikimedia-operations) [2022-06-09T07:13:32Z] <moritzm> drain ganeti3002 for firmware update/reimage T308238

MoritzMuehlenhoff reassigned this task from MoritzMuehlenhoff to RobH.Thu, Jun 9, 12:22 PM
MoritzMuehlenhoff updated the task description. (Show Details)

ganeti3002 is removed from the cluster, downtimed and needs the same firmware/NIC updates to enable the reimage to Bullseye.

Stashbot added a comment.Thu, Jun 9, 3:58 PM

Mentioned in SAL (#wikimedia-sre) [2022-06-09T15:58:38Z] <robh> ganeti3002 rebooting into firmware update then reimage via T308238

ops-monitoring-bot added a comment.Thu, Jun 9, 5:01 PM

Cookbook cookbooks.sre.hosts.reimage was started by robh@cumin1001 for host ganeti3002.esams.wmnet with OS bullseye

ops-monitoring-bot added a comment.Thu, Jun 9, 5:40 PM

Cookbook cookbooks.sre.hosts.reimage started by robh@cumin1001 for host ganeti3002.esams.wmnet with OS bullseye completed:

  • ganeti3002 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202206091701_robh_623293_ganeti3002.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
RobH reassigned this task from RobH to MoritzMuehlenhoff.Thu, Jun 9, 5:41 PM

ganeti3002 firmware updated for nic, bios, and idrac. reimaged and ready for next one after you juggle this back into service =]

MoritzMuehlenhoff updated the task description. (Show Details)Fri, Jun 10, 9:56 AM
Stashbot added a comment.Mon, Jun 13, 7:54 AM

Mentioned in SAL (#wikimedia-operations) [2022-06-13T07:54:58Z] <moritzm> failover ganeti master in esams to ganeti3003 T308238

MoritzMuehlenhoff updated the task description. (Show Details)Mon, Jun 13, 7:55 AM
Stashbot added a comment.Mon, Jun 13, 9:12 AM

Mentioned in SAL (#wikimedia-operations) [2022-06-13T09:12:16Z] <moritzm> drain ganeti3001 for firmware update/reimage T308238

MoritzMuehlenhoff reassigned this task from MoritzMuehlenhoff to RobH.Mon, Jun 13, 12:17 PM

ganeti3001 is removed from the cluster, downtimed and needs the same firmware/NIC updates to enable the reimage to Bullseye.

Stashbot added a comment.Mon, Jun 13, 4:10 PM

Mentioned in SAL (#wikimedia-sre) [2022-06-13T16:10:15Z] <robh> ganeti3001 rebooting and reimaging for firmware updates via T308238

ops-monitoring-bot added a comment.Mon, Jun 13, 4:32 PM

Cookbook cookbooks.sre.hosts.reimage was started by robh@cumin1001 for host ganeti3001.esams.wmnet with OS bullseye

ops-monitoring-bot added a comment.Mon, Jun 13, 5:11 PM

Cookbook cookbooks.sre.hosts.reimage started by robh@cumin1001 for host ganeti3001.esams.wmnet with OS bullseye completed:

  • ganeti3001 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202206131632_robh_1344098_ganeti3001.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
RobH reassigned this task from RobH to MoritzMuehlenhoff.Mon, Jun 13, 5:13 PM

ganeti3001 firmware updates
bios 2.2.11 to 2.14.2
nic 21.40.22.20 to 21.85.21.92
idrac 3.34.34.34 to 5.10.10.00

Moritz,

ganeti3001 firmware updated and reimaged to bullseye, back to you for service implementation

MoritzMuehlenhoff updated the task description. (Show Details)Tue, Jun 14, 10:32 AM
Stashbot added a comment.Tue, Jun 14, 11:02 AM

Mentioned in SAL (#wikimedia-operations) [2022-06-14T11:02:02Z] <moritzm> rebalancing ganeti cluster in esams T308238

MoritzMuehlenhoff closed this task as Resolved.Thu, Jun 16, 3:14 PM

This is complete.

Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL