- db1164 master
- db1217 backup source
- db2132 codfw master
- db2160 backup source
Description
Description
Details
Details
Related Changes in Gerrit:
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | Marostegui | T356960 Upgrade hosts to MariaDB 10.6 | |||
| Resolved | Marostegui | T349090 Migrate m1 to Debian Bookworm + MariaDB 10.6 | |||
| Resolved | Marostegui | T349165 Compile and package 10.6.15 for bookworm | |||
| Resolved | Marostegui | T349272 Move db1119 to m1 | |||
| Resolved | Marostegui | T350022 Switchover m1 master (db1164-> db1119) | |||
| Resolved | Marostegui | T351176 Switchover m1 master (db1119 -> db1164) |
Event Timeline
Comment Actions
Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1001 for host db2160.codfw.wmnet with OS bookworm
Comment Actions
db2160 is getting stuck on the following initial step of the Bookworm installer:
Loading debian-installer/amd64/initrd.gz...ing debian-installer/amd64/initrd.gz...
Comment Actions
Definitely not a one time thing, it's happened in all my attempts (3), @MoritzMuehlenhoff any thoughts?
For now I am going to reboot it and cancel the reimage
Comment Actions
Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1001 for host db2160.codfw.wmnet with OS bookworm executed with errors:
- db2160 (FAIL)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present and delete any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- The reimage failed, see the cookbook logs for the details
Comment Actions
Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1001 for host db2160.codfw.wmnet with OS bookworm
Comment Actions
Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1001 for host db2160.codfw.wmnet with OS bookworm executed with errors:
- db2160 (FAIL)
- Downtimed on Icinga/Alertmanager
- Unable to disable Puppet, the host may have been unreachable
- Removed from Puppet and PuppetDB if present and delete any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- The reimage failed, see the cookbook logs for the details
Comment Actions
Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1001 for host db2160.codfw.wmnet with OS bookworm
Comment Actions
Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1001 for host db2160.codfw.wmnet with OS bookworm completed:
- db2160 (WARN)
- Downtimed on Icinga/Alertmanager
- Unable to disable Puppet, the host may have been unreachable
- Removed from Puppet and PuppetDB if present and delete any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Add puppet_version metadata to Debian installer
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bookworm OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202310180639_marostegui_2482763_db2160.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is not optimal, downtime not removed
- Updated Netbox data from PuppetDB
Comment Actions
For codfw master, I am firstly going to migrate it to 10.6 and then to bookworm, I don't want to do both steps at the same time.
Comment Actions
Change 966806 had a related patch set uploaded (by Marostegui; author: Marostegui):
[operations/puppet@production] db2132: Migrate to 10.6
Comment Actions
Change 966806 merged by Marostegui:
[operations/puppet@production] db2132: Migrate to 10.6
Comment Actions
Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1001 for host db2132.codfw.wmnet with OS bookworm
Comment Actions
Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1001 for host db2132.codfw.wmnet with OS bookworm executed with errors:
- db2132 (FAIL)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present and delete any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Add puppet_version metadata to Debian installer
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bookworm OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202310180740_marostegui_2515814_db2132.out
- The reimage failed, see the cookbook logs for the details
Comment Actions
Change 969708 had a related patch set uploaded (by Marostegui; author: Marostegui):
[operations/puppet@production] db1217: Disable notifications
Comment Actions
Change 969708 merged by Marostegui:
[operations/puppet@production] db1217: Disable notifications
Comment Actions
Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1001 for host db1217.eqiad.wmnet with OS bookworm
Comment Actions
Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1001 for host db1217.eqiad.wmnet with OS bookworm completed:
- db1217 (WARN)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Add puppet_version metadata to Debian installer
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bookworm OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202310301226_marostegui_3452735_db1217.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is not optimal, downtime not removed
- Updated Netbox data from PuppetDB
Comment Actions
Change 974071 had a related patch set uploaded (by Marostegui; author: Marostegui):
[operations/puppet@production] db1164: Disable notifications
Comment Actions
Change 974071 merged by Marostegui:
[operations/puppet@production] db1164: Disable notifications
Comment Actions
The switchover is done. I am going to reimage db1164 and then I will switch back db1164 to become a master (this will be tracked in a different task)
Comment Actions
Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1001 for host db1164.eqiad.wmnet with OS bookworm
Comment Actions
Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1001 for host db1164.eqiad.wmnet with OS bookworm completed:
- db1164 (WARN)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Add puppet_version metadata to Debian installer
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bookworm OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202311140832_marostegui_418487_db1164.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is not optimal, downtime not removed
- Updated Netbox data from PuppetDB