- dbstore1008
- db2220
- db2218
- db2208
- db2200
- db2198
- db2187
- db2182
- db2168
- db2159 sanitarium master
- db2150
- db2122
- db2121 master
- db2120
- db2098 T362745 backup source to be decommissioned
- db1236
- db1227
- db1202
- db1194
- db1191
- db1181 master
- db1174
- db1171 backup source T362745
- db1170
- db1158 sanitarium master
- db1155
- clouddb1021
- clouddb1018
- clouddb1014
Description
Description
Details
Details
Related Changes in Gerrit:
Customize query in gerrit
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | Marostegui | T356960 Upgrade hosts to MariaDB 10.6 | |||
| Resolved | Marostegui | T362745 Upgrade s7 to MariaDB 10.6 | |||
| Resolved | Marostegui | T363668 Switchover s7 master (db2121 -> db2218) | |||
| Resolved | Marostegui | T363892 Switchover s7 master (db1181 -> db1236) |
Event Timeline
Comment Actions
Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1002 for host db2182.codfw.wmnet with OS bookworm
Comment Actions
Mentioned in SAL (#wikimedia-operations) [2024-04-17T04:59:58Z] <marostegui> dbmaint Upgrade s7 codfw to Bookworm and MariaDB 10.6 T362745
Comment Actions
Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1002 for host db2182.codfw.wmnet with OS bookworm completed:
- db2182 (PASS)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Add puppet_version metadata to Debian installer
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bookworm OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202404170515_marostegui_3082325_db2182.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB
Comment Actions
Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1002 for host db2150.codfw.wmnet with OS bookworm
Comment Actions
Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1002 for host db2150.codfw.wmnet with OS bookworm completed:
- db2150 (PASS)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Add puppet_version metadata to Debian installer
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bookworm OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202404170742_marostegui_3106077_db2150.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB
Comment Actions
Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1002 for host db2120.codfw.wmnet with OS bookworm
Comment Actions
Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1002 for host db2120.codfw.wmnet with OS bookworm completed:
- db2120 (PASS)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Add puppet_version metadata to Debian installer
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bookworm OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202404171244_marostegui_3164035_db2120.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB
Comment Actions
Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1002 for host db2108.codfw.wmnet with OS bookworm
Comment Actions
Mentioned in SAL (#wikimedia-operations) [2024-04-18T05:19:59Z] <marostegui> dbmaint Upgrade s7 codfw to Bookworm and MariaDB 10.6 T362745
Comment Actions
Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1002 for host db2108.codfw.wmnet with OS bookworm completed:
- db2108 (WARN)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Add puppet_version metadata to Debian installer
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bookworm OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202404180534_marostegui_3335067_db2108.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is not optimal, downtime not removed
- Updated Netbox data from PuppetDB
- Cleared switch DHCP cache and MAC table for the host IP and MAC (EVPN Switch)
Comment Actions
Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1002 for host db1202.eqiad.wmnet with OS bookworm
Comment Actions
Mentioned in SAL (#wikimedia-operations) [2024-04-19T05:02:17Z] <marostegui> dbmaint Upgrade s7 codfw to Bookworm and MariaDB 10.6 T362745
Comment Actions
Mentioned in SAL (#wikimedia-operations) [2024-04-19T05:02:25Z] <marostegui> dbmaint Upgrade s7 eqiad to Bookworm and MariaDB 10.6 T362745
Comment Actions
Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1002 for host db1202.eqiad.wmnet with OS bookworm completed:
- db1202 (PASS)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Add puppet_version metadata to Debian installer
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bookworm OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202404190506_marostegui_3557352_db1202.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB
- Cleared switch DHCP cache and MAC table for the host IP and MAC (EVPN Switch)
Comment Actions
Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1002 for host db1194.eqiad.wmnet with OS bookworm
Comment Actions
Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1002 for host db1194.eqiad.wmnet with OS bookworm completed:
- db1194 (WARN)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Add puppet_version metadata to Debian installer
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bookworm OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202404190755_marostegui_3580081_db1194.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is not optimal, downtime not removed
- Updated Netbox data from PuppetDB
- Cleared switch DHCP cache and MAC table for the host IP and MAC (EVPN Switch)
Comment Actions
Change #1025148 had a related patch set uploaded (by Marostegui; author: Marostegui):
[operations/puppet@production] db2159: Disable notifications
Comment Actions
Change #1025148 merged by Marostegui:
[operations/puppet@production] db2159: Disable notifications
Comment Actions
Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1002 for host db2159.codfw.wmnet with OS bookworm
Comment Actions
Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1002 for host db2159.codfw.wmnet with OS bookworm completed:
- db2159 (PASS)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Add puppet_version metadata to Debian installer
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bookworm OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202404290710_marostegui_2985134_db2159.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB
- Cleared switch DHCP cache and MAC table for the host IP and MAC (EVPN Switch)
Comment Actions
Change #1025281 had a related patch set uploaded (by Marostegui; author: Marostegui):
[operations/puppet@production] db2121: Disable notifications
Comment Actions
Change #1025281 merged by Marostegui:
[operations/puppet@production] db2121: Disable notifications
Comment Actions
Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1002 for host db2121.codfw.wmnet with OS bookworm
Comment Actions
Change #1025287 had a related patch set uploaded (by Marostegui; author: Marostegui):
[operations/puppet@production] db1191: Disable notifications
Comment Actions
Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1002 for host db2121.codfw.wmnet with OS bookworm completed:
- db2121 (PASS)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Add puppet_version metadata to Debian installer
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bookworm OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202404290947_marostegui_3010430_db2121.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB
- Cleared switch DHCP cache and MAC table for the host IP and MAC (EVPN Switch)
Comment Actions
Change #1025287 merged by Marostegui:
[operations/puppet@production] db1191: Disable notifications
Comment Actions
Mentioned in SAL (#wikimedia-operations) [2024-04-29T10:19:10Z] <marostegui@cumin1002> dbctl commit (dc=all): 'Depool db1191 T362745', diff saved to https://phabricator.wikimedia.org/P61373 and previous config saved to /var/cache/conftool/dbconfig/20240429-101908-marostegui.json
Comment Actions
Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1002 for host db1191.eqiad.wmnet with OS bookworm
Comment Actions
Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1002 for host db1191.eqiad.wmnet with OS bookworm completed:
- db1191 (WARN)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Add puppet_version metadata to Debian installer
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bookworm OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202404291037_marostegui_3023159_db1191.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is not optimal, downtime not removed
- Updated Netbox data from PuppetDB
- Cleared switch DHCP cache and MAC table for the host IP and MAC (EVPN Switch)
Comment Actions
Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1002 for host db1174.eqiad.wmnet with OS bookworm
Comment Actions
Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1002 for host db1174.eqiad.wmnet with OS bookworm completed:
- db1174 (PASS)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Add puppet_version metadata to Debian installer
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bookworm OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202404300704_marostegui_3204337_db1174.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB
Comment Actions
Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1002 for host db1158.eqiad.wmnet with OS bookworm
Comment Actions
Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1002 for host db1158.eqiad.wmnet with OS bookworm completed:
- db1158 (WARN)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Add puppet_version metadata to Debian installer
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bookworm OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202404300911_marostegui_3226358_db1158.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is not optimal, downtime not removed
- Updated Netbox data from PuppetDB
Comment Actions
Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1002 for host db1236.eqiad.wmnet with OS bookworm
Comment Actions
Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1002 for host db1236.eqiad.wmnet with OS bookworm completed:
- db1236 (PASS)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Add puppet_version metadata to Debian installer
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bookworm OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202405010514_marostegui_3695644_db1236.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB
Comment Actions
Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1002 for host db1181.eqiad.wmnet with OS bookworm
Comment Actions
Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1002 for host db1181.eqiad.wmnet with OS bookworm completed:
- db1181 (WARN)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Add puppet_version metadata to Debian installer
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bookworm OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202405020506_marostegui_3897616_db1181.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is not optimal, downtime not removed
- Updated Netbox data from PuppetDB
Comment Actions
Change #1035605 had a related patch set uploaded (by Marostegui; author: Marostegui):
[operations/puppet@production] db2122: Disable notifications
Comment Actions
Change #1035605 merged by Marostegui:
[operations/puppet@production] db2122: Disable notifications
Comment Actions
Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1002 for host db2122.codfw.wmnet with OS bookworm
Comment Actions
Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1002 for host db2122.codfw.wmnet with OS bookworm completed:
- db2122 (PASS)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Add puppet_version metadata to Debian installer
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bookworm OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202405240459_marostegui_2436214_db2122.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB
- Cleared switch DHCP cache and MAC table for the host IP and MAC (EVPN Switch)
Comment Actions
Change #1039178 had a related patch set uploaded (by Marostegui; author: Marostegui):
[operations/puppet@production] db1127: Disable notifications
Comment Actions
Change #1039178 merged by Marostegui:
[operations/puppet@production] db1227: Disable notifications
Comment Actions
Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1002 for host db1227.eqiad.wmnet with OS bookworm
Comment Actions
Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1002 for host db1227.eqiad.wmnet with OS bookworm completed:
- db1227 (PASS)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Add puppet_version metadata to Debian installer
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bookworm OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202406050902_marostegui_660747_db1227.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB