Description
Details
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | Marostegui | T422365 Migration to Debian Trixie of production database-related hosts | |||
| Resolved | Marostegui | T422368 Migrate parsercache to Debian Trixie | |||
| Unknown Object (Task) | |||||
| Resolved | Jhancock.wm | T418907 Q3:rack/setup/install pc202[1-4] | |||
| Unknown Object (Task) | |||||
| Resolved | VRiley-WMF | T418908 Q4:rack/setup/install pc102[1-4] | |||
| Resolved | Marostegui | T418973 Productionize pc20[21-24] and pc10[21-24] | |||
| Resolved | Request | Jhancock.wm | T424012 decommission pc2011.codfw.wmnet | ||
| Resolved | Request | Jhancock.wm | T424201 decommission pc2012.codfw.wmnet | ||
| Resolved | Request | Jhancock.wm | T426555 decommission pc2013.codfw.wmnet | ||
| Resolved | Request | Jhancock.wm | T426595 decommission pc2014.codfw.wmnet | ||
| Resolved | Request | VRiley-WMF | T426806 decommission pc1011.eqiad.wmnet | ||
| Resolved | Request | VRiley-WMF | T426930 decommission pc1012.eqiad.wmnet | ||
| Resolved | Request | Jclark-ctr | T427190 decommission pc1013.eqiad.wmnet | ||
| Resolved | Request | Jclark-ctr | T427270 decommission pc1014.eqiad.wmnet | ||
| Resolved | Marostegui | T429178 Upgrade pc7 and pc8 to Debian Trixie |
Event Timeline
Change #1268207 had a related patch set uploaded (by Marostegui; author: Marostegui):
[operations/puppet@production] pc5: Disable notifications
Depooled pc1015.eqiad.wmnet and pc2015.codfw.wmnet Migration to Debian Trixie - marostegui@cumin1003 - T422368
Change #1268207 merged by Marostegui:
[operations/puppet@production] pc5: Disable notifications
Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1003 for host pc1015.eqiad.wmnet with OS trixie
Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1003 for host pc2015.codfw.wmnet with OS trixie
Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1003 for host pc1015.eqiad.wmnet with OS trixie completed:
- pc1015 (PASS)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Checked BIOS boot parameters are back to normal
- Host up (new fresh trixie OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202604061149_marostegui_3102818_pc1015.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB
Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1003 for host pc2015.codfw.wmnet with OS trixie completed:
- pc2015 (PASS)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Checked BIOS boot parameters are back to normal
- Host up (new fresh trixie OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202604061153_marostegui_3102870_pc2015.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB
Depooled pc1015.eqiad.wmnet and pc2015.codfw.wmnet Migration to Debian Trixie - marostegui@cumin1003 - T422368
Depooled pc1011.eqiad.wmnet and pc2011.codfw.wmnet Upgrade package - marostegui@cumin1003 - T422368
Change #1269223 had a related patch set uploaded (by Marostegui; author: Marostegui):
[operations/puppet@production] pc2010: Disable notifications
Change #1269223 merged by Marostegui:
[operations/puppet@production] pc2016: Disable notifications
Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1003 for host pc2016.codfw.wmnet with OS trixie
Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1003 for host pc2016.codfw.wmnet with OS trixie completed:
- pc2016 (WARN)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Checked BIOS boot parameters are back to normal
- Host up (new fresh trixie OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202604090535_marostegui_3857939_pc2016.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is not optimal, downtime not removed
- Updated Netbox data from PuppetDB
Change #1269243 had a related patch set uploaded (by Marostegui; author: Marostegui):
[operations/puppet@production] pc1016: Disable notifications
Change #1269243 merged by Marostegui:
[operations/puppet@production] pc1016: Disable notifications
Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1003 for host pc1016.eqiad.wmnet with OS trixie
Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1003 for host pc1016.eqiad.wmnet with OS trixie executed with errors:
- pc1016 (FAIL)
- The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console pc1016.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1003 for host pc1016.eqiad.wmnet with OS trixie
Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1003 for host pc1016.eqiad.wmnet with OS trixie completed:
- pc1016 (PASS)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Checked BIOS boot parameters are back to normal
- Host up (new fresh trixie OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202604090659_marostegui_3870716_pc1016.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB
This is basically all done and waiting for the next HW to arrive (see subtasks) to get installed directly as Trixie