They already run 10.6, let's go to Debian Bookworm before they hit production
- pc1015
- pc1016
- pc2015
- pc2016
Description
Description
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Unknown Object (Task) | |||||
| Resolved | Jhancock.wm | T342163 Q1:rack/setup/install pc201[56] | |||
| Unknown Object (Task) | |||||
| Resolved | Jclark-ctr | T342164 Q1:rack/setup/install pc101[56] | |||
| Resolved | None | T343408 Productionize pc1015, p1016, pc2015 and pc2016 | |||
| Resolved | Marostegui | T350526 Upgrade pc4 to Debian Bookworm |
Event Timeline
Comment Actions
Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1001 for host pc2015.codfw.wmnet with OS bookworm
Comment Actions
Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1001 for host pc1015.eqiad.wmnet with OS bookworm
Comment Actions
Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1001 for host pc1015.eqiad.wmnet with OS bookworm completed:
- pc1015 (PASS)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Add puppet_version metadata to Debian installer
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bookworm OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202311061512_marostegui_4012722_pc1015.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB
Comment Actions
Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1001 for host pc2015.codfw.wmnet with OS bookworm completed:
- pc2015 (PASS)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Add puppet_version metadata to Debian installer
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bookworm OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202311061518_marostegui_4012396_pc2015.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB
Comment Actions
Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1001 for host pc2016.codfw.wmnet with OS bookworm
Comment Actions
Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1001 for host pc1016.eqiad.wmnet with OS bookworm
Comment Actions
Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1001 for host pc2016.codfw.wmnet with OS bookworm completed:
- pc2016 (PASS)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Add puppet_version metadata to Debian installer
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bookworm OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202311061552_marostegui_4033409_pc2016.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB
Comment Actions
Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1001 for host pc1016.eqiad.wmnet with OS bookworm completed:
- pc1016 (WARN)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Add puppet_version metadata to Debian installer
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bookworm OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Unable to downtime the new host on Icinga/Alertmanager, the sre.hosts.downtime cookbook returned 97
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202311061554_marostegui_4035431_pc1016.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB