Page MenuHomePhabricator
Create Task
Maniphest T296721

Migrate eqiad Ganeti cluster to Buster
Closed, ResolvedPublic
Actions

Description

  • Add component/ganeti216 to all nodes
  • Upgrade all nodes using "cumin A:ganeti-eqiad 'apt-get install -y ganeti'"
  • cumin A:ganeti-eqiad 'chown gnt-metad /var/log/ganeti/meta-daemon.log'
  • sudo gnt-cluster renew-crypto --new-cluster-certificate --new-rapi-certificate --new-spice-certificate
  • sudo gnt-cluster upgrade --to 2.16
  • sudo gnt-cluster renew-crypto --new-node-certificates (might not be needed for future updates if the hashing algo stays the same)

Following that empty/reimage/readd the following virtualisation servers:

  • ganeti1005.eqiad.wmnet
  • ganeti1006.eqiad.wmnet
  • ganeti1007.eqiad.wmnet
  • ganeti1008.eqiad.wmnet
  • ganeti1009.eqiad.wmnet
  • ganeti1010.eqiad.wmnet
  • ganeti1011.eqiad.wmnet
  • ganeti1012.eqiad.wmnet
  • ganeti1013.eqiad.wmnet
  • ganeti1014.eqiad.wmnet
  • ganeti1015.eqiad.wmnet
  • ganeti1016.eqiad.wmnet
  • ganeti1017.eqiad.wmnet
  • ganeti1018.eqiad.wmnet
  • ganeti1019.eqiad.wmnet
  • ganeti1020.eqiad.wmnet
  • ganeti1021.eqiad.wmnet
  • ganeti1022.eqiad.wmnet

The following instances were temporarily converted from plain disk storage to DRBD:

  • kubestagetcd1004 (converted back to plain)
  • kubestagetcd1005 (converted back to plain)
  • kubestagetcd1006 (converted back to plain)
  • kubetcd1004 (converted back to plain)
  • kubetcd1005 (converted back to plain)
  • kubetcd1006 (converted back to plain)
  • ml-etcd1001
  • ml-etcd1002
  • ml-etcd1003

Details

SubjectRepoBranchLines +/-
Enable ganeti 2.16 in eqiadoperations/puppetproduction+2 -0
Point back irc.wikimedia.org to irc2001operations/dnsmaster+1 -1
Customize query in gerrit

Related Objects
Search...

Event Timeline

MoritzMuehlenhoff created this task.Nov 30 2021, 12:11 PM
gerritbot added a comment.Nov 30 2021, 2:36 PM

Change 742730 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/dns@master] Point back irc.wikimedia.org to irc2001

https://gerrit.wikimedia.org/r/742730

gerritbot added a project: Patch-For-Review.Nov 30 2021, 2:36 PM
gerritbot added a comment.Dec 1 2021, 11:59 AM

Change 742730 merged by Muehlenhoff:

[operations/dns@master] Point back irc.wikimedia.org to irc2001

https://gerrit.wikimedia.org/r/742730

Maintenance_bot removed a project: Patch-For-Review.Dec 1 2021, 12:10 PM
herron triaged this task as Medium priority.Dec 3 2021, 7:08 PM
MoritzMuehlenhoff updated the task description. (Show Details)Jan 12 2022, 4:39 PM
MoritzMuehlenhoff updated the task description. (Show Details)Jan 14 2022, 11:15 AM
MoritzMuehlenhoff updated the task description. (Show Details)Jan 17 2022, 11:23 AM
gerritbot added a comment.Jan 17 2022, 4:29 PM

Change 754540 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Enable ganeti 2.16 in eqiad

https://gerrit.wikimedia.org/r/754540

gerritbot added a project: Patch-For-Review.Jan 17 2022, 4:29 PM
gerritbot added a comment.Jan 18 2022, 8:19 AM

Change 754540 merged by Muehlenhoff:

[operations/puppet@production] Enable ganeti 2.16 in eqiad

https://gerrit.wikimedia.org/r/754540

MoritzMuehlenhoff updated the task description. (Show Details)Jan 18 2022, 9:11 AM
Stashbot added a comment.Jan 18 2022, 9:50 AM

Mentioned in SAL (#wikimedia-operations) [2022-01-18T09:50:01Z] <moritzm> installing ganeti 2.16.0-1~bpo9+1+wmf1 on ganeti/eqiad servers T296721

MoritzMuehlenhoff updated the task description. (Show Details)Jan 18 2022, 9:56 AM
MoritzMuehlenhoff updated the task description. (Show Details)Jan 18 2022, 10:32 AM
MoritzMuehlenhoff updated the task description. (Show Details)Jan 18 2022, 11:19 AM
ops-monitoring-bot added a comment.Jan 19 2022, 2:35 PM

Cookbook cookbooks.sre.hosts.reimage was started by jmm@cumin2002 for host ganeti1018.eqiad.wmnet with OS buster

ops-monitoring-bot added a comment.Jan 19 2022, 3:00 PM

Cookbook cookbooks.sre.hosts.reimage started by jmm@cumin2002 for host ganeti1018.eqiad.wmnet with OS buster executed with errors:

  • ganeti1018 (FAIL)
    • Downtimed on Icinga
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Jan 20 2022, 9:50 AM

Cookbook cookbooks.sre.hosts.reimage was started by jmm@cumin2002 for host ganeti1018.eqiad.wmnet with OS buster

ops-monitoring-bot added a comment.Jan 20 2022, 10:45 AM

Cookbook cookbooks.sre.hosts.reimage started by jmm@cumin2002 for host ganeti1018.eqiad.wmnet with OS buster completed:

  • ganeti1018 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh buster OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202201200950_jmm_3211404_ganeti1018.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
MoritzMuehlenhoff updated the task description. (Show Details)Jan 21 2022, 10:26 AM
ops-monitoring-bot added a comment.Jan 25 2022, 8:44 AM

Cookbook cookbooks.sre.hosts.reimage was started by jmm@cumin2002 for host ganeti1013.eqiad.wmnet with OS buster

ops-monitoring-bot added a comment.Jan 25 2022, 9:11 AM

Cookbook cookbooks.sre.hosts.reimage started by jmm@cumin2002 for host ganeti1013.eqiad.wmnet with OS buster completed:

  • ganeti1013 (PASS)
    • Downtimed on Icinga
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh buster OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202201250843_jmm_4048169_ganeti1013.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
MoritzMuehlenhoff updated the task description. (Show Details)Jan 26 2022, 8:20 AM
ops-monitoring-bot added a comment.Jan 26 2022, 8:29 AM

Cookbook cookbooks.sre.hosts.reimage was started by jmm@cumin2002 for host ganeti1014.eqiad.wmnet with OS buster

ops-monitoring-bot added a comment.Jan 26 2022, 8:56 AM

Cookbook cookbooks.sre.hosts.reimage started by jmm@cumin2002 for host ganeti1014.eqiad.wmnet with OS buster completed:

  • ganeti1014 (PASS)
    • Downtimed on Icinga
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh buster OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202201260829_jmm_21158_ganeti1014.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Jan 26 2022, 9:00 AM

Cookbook cookbooks.sre.hosts.reimage was started by jmm@cumin2002 for host ganeti1005.eqiad.wmnet with OS buster

ops-monitoring-bot added a comment.Jan 26 2022, 9:25 AM

Cookbook cookbooks.sre.hosts.reimage started by jmm@cumin2002 for host ganeti1005.eqiad.wmnet with OS buster executed with errors:

  • ganeti1005 (FAIL)
    • Downtimed on Icinga
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Jan 26 2022, 9:33 AM

Cookbook cookbooks.sre.hosts.reimage was started by jmm@cumin2002 for host ganeti1005.eqiad.wmnet with OS buster

ops-monitoring-bot added a comment.Jan 26 2022, 9:47 AM

Cookbook cookbooks.sre.hosts.reimage started by jmm@cumin2002 for host ganeti1005.eqiad.wmnet with OS buster completed:

  • ganeti1005 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202201260932_jmm_30181_ganeti1005.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Jan 26 2022, 9:57 AM

Cookbook cookbooks.sre.hosts.reimage was started by jmm@cumin2002 for host ganeti1006.eqiad.wmnet with OS buster

ops-monitoring-bot added a comment.Jan 26 2022, 10:25 AM

Cookbook cookbooks.sre.hosts.reimage started by jmm@cumin2002 for host ganeti1006.eqiad.wmnet with OS buster completed:

  • ganeti1006 (PASS)
    • Downtimed on Icinga
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh buster OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202201260957_jmm_34636_ganeti1006.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
MoritzMuehlenhoff updated the task description. (Show Details)Jan 26 2022, 1:56 PM
MoritzMuehlenhoff updated the task description. (Show Details)Jan 26 2022, 2:56 PM
MoritzMuehlenhoff updated the task description. (Show Details)Jan 26 2022, 3:12 PM
ops-monitoring-bot added a comment.Jan 31 2022, 9:21 AM

Cookbook cookbooks.sre.hosts.reimage was started by jmm@cumin2002 for host ganeti1007.eqiad.wmnet with OS buster

ops-monitoring-bot added a comment.Jan 31 2022, 9:45 AM

Cookbook cookbooks.sre.hosts.reimage started by jmm@cumin2002 for host ganeti1007.eqiad.wmnet with OS buster completed:

  • ganeti1007 (PASS)
    • Downtimed on Icinga
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh buster OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202201310921_jmm_855514_ganeti1007.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Jan 31 2022, 10:01 AM

Cookbook cookbooks.sre.hosts.reimage was started by jmm@cumin2002 for host ganeti1015.eqiad.wmnet with OS buster

ops-monitoring-bot added a comment.Jan 31 2022, 10:42 AM

Cookbook cookbooks.sre.hosts.reimage started by jmm@cumin2002 for host ganeti1015.eqiad.wmnet with OS buster completed:

  • ganeti1015 (PASS)
    • Downtimed on Icinga
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh buster OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202201311001_jmm_860999_ganeti1015.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
MoritzMuehlenhoff updated the task description. (Show Details)Jan 31 2022, 1:16 PM
MoritzMuehlenhoff updated the task description. (Show Details)Jan 31 2022, 2:16 PM
ops-monitoring-bot added a comment.Feb 1 2022, 9:58 AM

Cookbook cookbooks.sre.hosts.reimage was started by jmm@cumin2002 for host ganeti1010.eqiad.wmnet with OS buster

ops-monitoring-bot added a comment.Feb 1 2022, 10:24 AM

Cookbook cookbooks.sre.hosts.reimage started by jmm@cumin2002 for host ganeti1010.eqiad.wmnet with OS buster completed:

  • ganeti1010 (PASS)
    • Downtimed on Icinga
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh buster OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202202010957_jmm_1040637_ganeti1010.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
MoritzMuehlenhoff updated the task description. (Show Details)Feb 1 2022, 3:39 PM
ops-monitoring-bot added a comment.Feb 2 2022, 9:17 AM

Cookbook cookbooks.sre.hosts.reimage was started by jmm@cumin2002 for host ganeti1019.eqiad.wmnet with OS buster

ops-monitoring-bot added a comment.Feb 2 2022, 9:47 AM

Cookbook cookbooks.sre.hosts.reimage started by jmm@cumin2002 for host ganeti1019.eqiad.wmnet with OS buster completed:

  • ganeti1019 (PASS)
    • Downtimed on Icinga
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh buster OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202202020917_jmm_1219045_ganeti1019.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Feb 2 2022, 9:53 AM

Cookbook cookbooks.sre.hosts.reimage was started by jmm@cumin2002 for host ganeti1008.eqiad.wmnet with OS buster

ops-monitoring-bot added a comment.Feb 2 2022, 10:21 AM

Cookbook cookbooks.sre.hosts.reimage started by jmm@cumin2002 for host ganeti1008.eqiad.wmnet with OS buster completed:

  • ganeti1008 (PASS)
    • Downtimed on Icinga
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh buster OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202202020952_jmm_1224361_ganeti1008.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
MoritzMuehlenhoff updated the task description. (Show Details)Feb 2 2022, 12:33 PM
MoritzMuehlenhoff updated the task description. (Show Details)Feb 4 2022, 10:07 AM
Stashbot added a comment.Feb 4 2022, 10:40 AM

Mentioned in SAL (#wikimedia-traffic) [2022-02-04T10:40:43Z] <moritzm> rebalancing row A in ganeti/eqiad, all nodes of that row are now running Buster T296721

ops-monitoring-bot added a comment.Feb 4 2022, 11:14 AM

Cookbook cookbooks.sre.hosts.reimage was started by jmm@cumin2002 for host ganeti1020.eqiad.wmnet with OS buster

ops-monitoring-bot added a comment.Feb 4 2022, 12:08 PM

Cookbook cookbooks.sre.hosts.reimage started by jmm@cumin2002 for host ganeti1020.eqiad.wmnet with OS buster completed:

  • ganeti1020 (PASS)
    • Downtimed on Icinga
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh buster OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202202041114_jmm_1577275_ganeti1020.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
MoritzMuehlenhoff updated the task description. (Show Details)Feb 7 2022, 12:39 PM
MoritzMuehlenhoff updated the task description. (Show Details)Feb 7 2022, 1:19 PM
ops-monitoring-bot added a comment.Feb 8 2022, 11:59 AM

Cookbook cookbooks.sre.hosts.reimage was started by jmm@cumin2002 for host ganeti1011.eqiad.wmnet with OS buster

ops-monitoring-bot added a comment.Feb 8 2022, 12:27 PM

Cookbook cookbooks.sre.hosts.reimage started by jmm@cumin2002 for host ganeti1011.eqiad.wmnet with OS buster executed with errors:

  • ganeti1011 (FAIL)
    • Downtimed on Icinga
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Feb 11 2022, 8:36 AM

Cookbook cookbooks.sre.hosts.reimage was started by jmm@cumin2002 for host ganeti1011.eqiad.wmnet with OS buster

ops-monitoring-bot added a comment.Feb 11 2022, 8:57 AM

Cookbook cookbooks.sre.hosts.reimage started by jmm@cumin2002 for host ganeti1011.eqiad.wmnet with OS buster executed with errors:

  • ganeti1011 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Feb 11 2022, 10:11 AM

Cookbook cookbooks.sre.hosts.reimage was started by jmm@cumin2002 for host ganeti1021.eqiad.wmnet with OS buster

ops-monitoring-bot added a comment.Feb 11 2022, 10:39 AM

Cookbook cookbooks.sre.hosts.reimage started by jmm@cumin2002 for host ganeti1021.eqiad.wmnet with OS buster completed:

  • ganeti1021 (PASS)
    • Downtimed on Icinga
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh buster OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202202111011_jmm_2798436_ganeti1021.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Feb 11 2022, 12:13 PM

Cookbook cookbooks.sre.hosts.reimage was started by jmm@cumin2002 for host ganeti1016.eqiad.wmnet with OS buster

ops-monitoring-bot added a comment.Feb 11 2022, 12:41 PM

Cookbook cookbooks.sre.hosts.reimage started by jmm@cumin2002 for host ganeti1016.eqiad.wmnet with OS buster completed:

  • ganeti1016 (PASS)
    • Downtimed on Icinga
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh buster OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202202111213_jmm_2813492_ganeti1016.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Feb 11 2022, 12:53 PM

Cookbook cookbooks.sre.hosts.reimage was started by jmm@cumin2002 for host ganeti1011.eqiad.wmnet with OS buster

ops-monitoring-bot added a comment.Feb 11 2022, 1:19 PM

Cookbook cookbooks.sre.hosts.reimage started by jmm@cumin2002 for host ganeti1011.eqiad.wmnet with OS buster executed with errors:

  • ganeti1011 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • The reimage failed, see the cookbook logs for the details
MoritzMuehlenhoff updated the task description. (Show Details)Feb 14 2022, 11:51 AM
MoritzMuehlenhoff updated the task description. (Show Details)Feb 14 2022, 12:48 PM
MoritzMuehlenhoff updated the task description. (Show Details)Feb 16 2022, 11:57 AM
Stashbot added a comment.Feb 16 2022, 2:17 PM

Mentioned in SAL (#wikimedia-operations) [2022-02-16T14:17:52Z] <moritzm> failover the ganeti master to ganeti1024 T296721

ops-monitoring-bot added a comment.Feb 18 2022, 8:45 AM

Cookbook cookbooks.sre.hosts.reimage was started by jmm@cumin2002 for host ganeti1022.eqiad.wmnet with OS buster

ops-monitoring-bot added a comment.Feb 18 2022, 9:33 AM

Cookbook cookbooks.sre.hosts.reimage started by jmm@cumin2002 for host ganeti1022.eqiad.wmnet with OS buster completed:

  • ganeti1022 (PASS)
    • Downtimed on Icinga
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh buster OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202202180844_jmm_4001000_ganeti1022.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Feb 18 2022, 11:27 AM

Cookbook cookbooks.sre.hosts.reimage was started by jmm@cumin2002 for host ganeti1017.eqiad.wmnet with OS buster

ops-monitoring-bot added a comment.Feb 18 2022, 11:54 AM

Cookbook cookbooks.sre.hosts.reimage started by jmm@cumin2002 for host ganeti1017.eqiad.wmnet with OS buster completed:

  • ganeti1017 (PASS)
    • Downtimed on Icinga
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh buster OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202202181127_jmm_4021290_ganeti1017.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Feb 18 2022, 1:12 PM

Cookbook cookbooks.sre.hosts.reimage was started by jmm@cumin2002 for host ganeti1012.eqiad.wmnet with OS buster

ops-monitoring-bot added a comment.Feb 18 2022, 1:41 PM

Cookbook cookbooks.sre.hosts.reimage started by jmm@cumin2002 for host ganeti1012.eqiad.wmnet with OS buster completed:

  • ganeti1012 (PASS)
    • Downtimed on Icinga
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh buster OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202202181312_jmm_4035993_ganeti1012.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Feb 21 2022, 8:50 AM

Cookbook cookbooks.sre.hosts.reimage was started by jmm@cumin2002 for host ganeti1009.eqiad.wmnet with OS buster

ops-monitoring-bot added a comment.Feb 21 2022, 9:33 AM

Cookbook cookbooks.sre.hosts.reimage started by jmm@cumin2002 for host ganeti1009.eqiad.wmnet with OS buster completed:

  • ganeti1009 (PASS)
    • Downtimed on Icinga
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh buster OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202202210849_jmm_322514_ganeti1009.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
MoritzMuehlenhoff updated the task description. (Show Details)Feb 21 2022, 10:58 AM
MoritzMuehlenhoff updated the task description. (Show Details)Feb 21 2022, 11:27 AM
MoritzMuehlenhoff updated the task description. (Show Details)Feb 21 2022, 12:16 PM
MoritzMuehlenhoff updated the task description. (Show Details)Feb 21 2022, 1:03 PM
Stashbot added a comment.Feb 21 2022, 1:06 PM

Mentioned in SAL (#wikimedia-operations) [2022-02-21T13:06:56Z] <moritzm> rebalance ganeti row_C (add nodes reimaged in there) T296721

Stashbot added a comment.Feb 22 2022, 1:24 PM

Mentioned in SAL (#wikimedia-operations) [2022-02-22T13:24:10Z] <moritzm> rebalance ganeti eqiad row_D (all nodes reimaged in there) T296721

Stashbot added a comment.Feb 22 2022, 6:30 PM

Mentioned in SAL (#wikimedia-operations) [2022-02-22T18:30:49Z] <moritzm> rebalance ganeti eqiad row_B (all nodes reimaged in there) T296721

MoritzMuehlenhoff closed this task as Resolved.Feb 23 2022, 9:08 AM
MoritzMuehlenhoff claimed this task.

The update is complete

lmata awarded a token.Feb 23 2022, 11:34 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL