⚓ T331713 Migrate restbase servers to Bullseye

Subject	Repo	Branch	Lines +/-
install_server: utilize reuse recipe for restbase2027	operations/puppet	production	+1 -2
install_server: add restbase1029 (as 3-ssd reuse)	operations/puppet	production	+1 -1
install: Use from-scratch partman recipe for restbase1030	operations/puppet	production	+2 -1

Status	Subtype	Assigned	Task
Open		None	T291916 Tracking task for Bullseye migrations in production
Resolved		Eevans	T331713 Migrate restbase servers to Bullseye
Resolved		Eevans	T346803 Unable to bootstrap restbase1030-{a,b,c}
Resolved		Eevans	T328490 restbase cluster: decommission end-of-life hosts
Resolved	Request	Jhancock.wm	T349526 decommission restbase2012.codfw.wmnet
Resolved	Request	Jclark-ctr	T349711 decommission restbase1018
Resolved	Request	Jclark-ctr	T349710 decommission restbase1017
Resolved	Request	Jclark-ctr	T349709 decommission restbase1016

Cookbook cookbooks.sre.hosts.reimage started by eevans@cumin1001 for host restbase1019.eqiad.wmnet with OS bullseye completed:

restbase1019 (WARN)
- Removed from Puppet and PuppetDB if present
- Deleted any existing Puppet certificate
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bullseye OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202309271658_eevans_668600_restbase1019.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is not optimal, downtime not removed
- Updated Netbox data from PuppetDB

Eevans updated the task description. (Show Details)Sep 27 2023, 5:38 PM

Cookbook cookbooks.sre.hosts.reimage was started by eevans@cumin1001 for host restbase2026.codfw.wmnet with OS bullseye

Cookbook cookbooks.sre.hosts.reimage started by eevans@cumin1001 for host restbase2026.codfw.wmnet with OS bullseye completed:

restbase2026 (PASS)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present
- Deleted any existing Puppet certificate
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bullseye OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202309271827_eevans_712875_restbase2026.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB

Eevans claimed this task.Sep 27 2023, 6:48 PM

Eevans updated the task description. (Show Details)

Cookbook cookbooks.sre.hosts.reimage was started by eevans@cumin1001 for host restbase2027.codfw.wmnet with OS bullseye

Cookbook cookbooks.sre.hosts.reimage started by eevans@cumin1001 for host restbase2027.codfw.wmnet with OS bullseye completed:

restbase2027 (WARN)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present
- Deleted any existing Puppet certificate
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bullseye OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202309271924_eevans_742451_restbase2027.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is not optimal, downtime not removed
- Updated Netbox data from PuppetDB

Eevans updated the task description. (Show Details)Sep 28 2023, 9:28 PM

Eevans updated the task description. (Show Details)Sep 28 2023, 9:41 PM

Eevans updated the task description. (Show Details)Sep 28 2023, 9:53 PM

Eevans updated the task description. (Show Details)Sep 28 2023, 9:59 PM

Eevans updated the task description. (Show Details)Sep 29 2023, 2:53 PM

Eevans updated the task description. (Show Details)

Change 962048 had a related patch set uploaded (by Eevans; author: Eevans):

[operations/puppet@production] install_server: utilize reuse recipe for restbase2027

https://gerrit.wikimedia.org/r/962048

gerritbot added a project: Patch-For-Review.Sep 29 2023, 3:05 PM

Eevans updated the task description. (Show Details)Sep 29 2023, 3:18 PM

Eevans updated the task description. (Show Details)Sep 29 2023, 3:34 PM

Eevans updated the task description. (Show Details)Sep 29 2023, 3:48 PM

Eevans updated the task description. (Show Details)Sep 29 2023, 4:07 PM

Eevans updated the task description. (Show Details)Sep 29 2023, 5:52 PM

Eevans updated the task description. (Show Details)Sep 29 2023, 6:17 PM

Eevans updated the task description. (Show Details)Sep 29 2023, 7:00 PM

Cookbook cookbooks.sre.hosts.reimage was started by eevans@cumin1001 for host restbase1020.eqiad.wmnet with OS bullseye

Cookbook cookbooks.sre.hosts.reimage started by eevans@cumin1001 for host restbase1020.eqiad.wmnet with OS bullseye completed:

restbase1020 (PASS)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present
- Deleted any existing Puppet certificate
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bullseye OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202310021354_eevans_4095129_restbase1020.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB

Eevans updated the task description. (Show Details)Oct 2 2023, 2:19 PM

Cookbook cookbooks.sre.hosts.reimage was started by eevans@cumin1001 for host restbase1021.eqiad.wmnet with OS bullseye

Cookbook cookbooks.sre.hosts.reimage started by eevans@cumin1001 for host restbase1021.eqiad.wmnet with OS bullseye completed:

restbase1021 (WARN)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present
- Deleted any existing Puppet certificate
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bullseye OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202310021437_eevans_4116560_restbase1021.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is not optimal, downtime not removed
- Updated Netbox data from PuppetDB

Eevans updated the task description. (Show Details)Oct 2 2023, 3:26 PM

Cookbook cookbooks.sre.hosts.reimage was started by eevans@cumin1001 for host restbase1028.eqiad.wmnet with OS bullseye

Cookbook cookbooks.sre.hosts.reimage started by eevans@cumin1001 for host restbase1028.eqiad.wmnet with OS bullseye completed:

restbase1028 (PASS)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present
- Deleted any existing Puppet certificate
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bullseye OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202310021543_eevans_4152248_restbase1028.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB

Eevans updated the task description. (Show Details)Oct 2 2023, 4:10 PM

Cookbook cookbooks.sre.hosts.reimage was started by eevans@cumin1001 for host restbase1031.eqiad.wmnet with OS bullseye

Cookbook cookbooks.sre.hosts.reimage started by eevans@cumin1001 for host restbase1031.eqiad.wmnet with OS bullseye completed:

restbase1031 (PASS)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present
- Deleted any existing Puppet certificate
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bullseye OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202310021629_eevans_4175765_restbase1031.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB

Cookbook cookbooks.sre.hosts.reimage was started by eevans@cumin1001 for host restbase1022.eqiad.wmnet with OS bullseye

Cookbook cookbooks.sre.hosts.reimage started by eevans@cumin1001 for host restbase1022.eqiad.wmnet with OS bullseye completed:

restbase1022 (WARN)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present
- Deleted any existing Puppet certificate
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bullseye OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202310021712_eevans_5464_restbase1022.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is not optimal, downtime not removed
- Updated Netbox data from PuppetDB

Eevans updated the task description. (Show Details)Oct 2 2023, 5:58 PM

Cookbook cookbooks.sre.hosts.reimage was started by eevans@cumin1001 for host restbase1023.eqiad.wmnet with OS bullseye

Cookbook cookbooks.sre.hosts.reimage started by eevans@cumin1001 for host restbase1023.eqiad.wmnet with OS bullseye completed:

restbase1023 (WARN)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present
- Deleted any existing Puppet certificate
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bullseye OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202310021816_eevans_37676_restbase1023.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is not optimal, downtime not removed
- Updated Netbox data from PuppetDB

Eevans updated the task description. (Show Details)Oct 2 2023, 6:42 PM

Eevans updated the task description. (Show Details)Oct 2 2023, 6:54 PM

Eevans updated the task description. (Show Details)Oct 2 2023, 7:01 PM

Cookbook cookbooks.sre.hosts.reimage was started by eevans@cumin1001 for host restbase1024.eqiad.wmnet with OS bullseye

Eevans updated the task description. (Show Details)Oct 2 2023, 7:13 PM

Cookbook cookbooks.sre.hosts.reimage started by eevans@cumin1001 for host restbase1024.eqiad.wmnet with OS bullseye completed:

restbase1024 (PASS)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present
- Deleted any existing Puppet certificate
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bullseye OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202310021919_eevans_69932_restbase1024.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB

Eevans updated the task description. (Show Details)Oct 2 2023, 7:52 PM

Cookbook cookbooks.sre.hosts.reimage was started by eevans@cumin1001 for host restbase1029.eqiad.wmnet with OS bullseye

Cookbook cookbooks.sre.hosts.reimage started by eevans@cumin1001 for host restbase1029.eqiad.wmnet with OS bullseye executed with errors:

restbase1029 (FAIL)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present
- Deleted any existing Puppet certificate
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- The reimage failed, see the cookbook logs for the details

Cookbook cookbooks.sre.hosts.reimage was started by eevans@cumin1001 for host restbase1029.eqiad.wmnet with OS bullseye

Eevans updated the task description. (Show Details)Oct 2 2023, 8:04 PM

Cookbook cookbooks.sre.hosts.reimage started by eevans@cumin1001 for host restbase1029.eqiad.wmnet with OS bullseye executed with errors:

restbase1029 (FAIL)
- Removed from Puppet and PuppetDB if present
- Deleted any existing Puppet certificate
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Checked BIOS boot parameters are back to normal
- The reimage failed, see the cookbook logs for the details

Change 962693 had a related patch set uploaded (by Eevans; author: Eevans):

[operations/puppet@production] install_server: add restbase1029 (as 3-ssd reuse)

https://gerrit.wikimedia.org/r/962693

Change 962693 merged by Eevans:

[operations/puppet@production] install_server: add restbase1029 (as 3-ssd reuse)

https://gerrit.wikimedia.org/r/962693

Cookbook cookbooks.sre.hosts.reimage was started by eevans@cumin1001 for host restbase1029.eqiad.wmnet with OS bullseye

Cookbook cookbooks.sre.hosts.reimage started by eevans@cumin1001 for host restbase1029.eqiad.wmnet with OS bullseye executed with errors:

restbase1029 (FAIL)
- Removed from Puppet and PuppetDB if present
- Deleted any existing Puppet certificate
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Checked BIOS boot parameters are back to normal
- The reimage failed, see the cookbook logs for the details

Cookbook cookbooks.sre.hosts.reimage was started by eevans@cumin1001 for host restbase1029.eqiad.wmnet with OS bullseye

Cookbook cookbooks.sre.hosts.reimage started by eevans@cumin1001 for host restbase1029.eqiad.wmnet with OS bullseye executed with errors:

restbase1029 (FAIL)
- Removed from Puppet and PuppetDB if present
- Deleted any existing Puppet certificate
- Removed from Debmonitor if present
- The reimage failed, see the cookbook logs for the details

Cookbook cookbooks.sre.hosts.reimage was started by eevans@cumin1001 for host restbase1029.eqiad.wmnet with OS bullseye

Cookbook cookbooks.sre.hosts.reimage started by eevans@cumin1001 for host restbase1029.eqiad.wmnet with OS bullseye executed with errors:

restbase1029 (FAIL)
- Removed from Puppet and PuppetDB if present
- Deleted any existing Puppet certificate
- Removed from Debmonitor if present
- The reimage failed, see the cookbook logs for the details

Cookbook cookbooks.sre.hosts.reimage was started by eevans@cumin1001 for host restbase1029.eqiad.wmnet with OS bullseye

Cookbook cookbooks.sre.hosts.reimage started by eevans@cumin1001 for host restbase1029.eqiad.wmnet with OS bullseye executed with errors:

restbase1029 (FAIL)
- Removed from Puppet and PuppetDB if present
- Deleted any existing Puppet certificate
- Removed from Debmonitor if present
- The reimage failed, see the cookbook logs for the details

Cookbook cookbooks.sre.hosts.reimage was started by eevans@cumin1001 for host restbase1029.eqiad.wmnet with OS bullseye

Eevans updated the task description. (Show Details)Oct 2 2023, 9:19 PM

Cookbook cookbooks.sre.hosts.reimage started by eevans@cumin1001 for host restbase1029.eqiad.wmnet with OS bullseye completed:

restbase1029 (PASS)
- Removed from Puppet and PuppetDB if present
- Deleted any existing Puppet certificate
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bullseye OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202310022057_eevans_119030_restbase1029.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB

Cookbook cookbooks.sre.hosts.reimage was started by eevans@cumin1001 for host restbase1032.eqiad.wmnet with OS bullseye

Cookbook cookbooks.sre.hosts.reimage started by eevans@cumin1001 for host restbase1032.eqiad.wmnet with OS bullseye executed with errors:

restbase1032 (FAIL)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present
- Deleted any existing Puppet certificate
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Checked BIOS boot parameters are back to normal
- The reimage failed, see the cookbook logs for the details

Cookbook cookbooks.sre.hosts.reimage was started by eevans@cumin1001 for host restbase1032.eqiad.wmnet with OS bullseye

Cookbook cookbooks.sre.hosts.reimage started by eevans@cumin1001 for host restbase1032.eqiad.wmnet with OS bullseye completed:

restbase1032 (PASS)
- Removed from Puppet and PuppetDB if present
- Deleted any existing Puppet certificate
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bullseye OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202310022246_eevans_173583_restbase1032.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB

Eevans updated the task description. (Show Details)Oct 2 2023, 11:14 PM

Eevans updated the task description. (Show Details)Oct 3 2023, 12:57 AM

Eevans updated the task description. (Show Details)Oct 3 2023, 1:15 AM

Eevans updated the task description. (Show Details)

Eevans updated the task description. (Show Details)Oct 3 2023, 1:32 AM

Eevans updated the task description. (Show Details)Oct 3 2023, 1:53 AM

Cookbook cookbooks.sre.hosts.reimage was started by eevans@cumin1001 for host restbase1025.eqiad.wmnet with OS bullseye

Cookbook cookbooks.sre.hosts.reimage started by eevans@cumin1001 for host restbase1025.eqiad.wmnet with OS bullseye completed:

restbase1025 (WARN)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present
- Deleted any existing Puppet certificate
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bullseye OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202310031307_eevans_593083_restbase1025.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is not optimal, downtime not removed
- Updated Netbox data from PuppetDB

Eevans updated the task description. (Show Details)Oct 3 2023, 1:42 PM

Cookbook cookbooks.sre.hosts.reimage was started by eevans@cumin1001 for host restbase1033.eqiad.wmnet with OS bullseye

Eevans updated the task description. (Show Details)Oct 3 2023, 1:46 PM

Cookbook cookbooks.sre.hosts.reimage started by eevans@cumin1001 for host restbase1033.eqiad.wmnet with OS bullseye completed:

restbase1033 (WARN)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present
- Deleted any existing Puppet certificate
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bullseye OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202310031401_eevans_623576_restbase1033.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is not optimal, downtime not removed
- Updated Netbox data from PuppetDB

Eevans updated the task description. (Show Details)Oct 3 2023, 3:10 PM

Eevans updated the task description. (Show Details)Oct 3 2023, 3:23 PM

Cookbook cookbooks.sre.hosts.reimage was started by eevans@cumin1001 for host restbase1026.eqiad.wmnet with OS bullseye

Eevans updated the task description. (Show Details)Oct 3 2023, 4:07 PM

Cookbook cookbooks.sre.hosts.reimage started by eevans@cumin1001 for host restbase1026.eqiad.wmnet with OS bullseye completed:

restbase1026 (PASS)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present
- Deleted any existing Puppet certificate
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bullseye OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202310031540_eevans_677743_restbase1026.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB

Eevans added a subtask: T328490: restbase cluster: decommission end-of-life hosts.Oct 3 2023, 4:27 PM

Cookbook cookbooks.sre.hosts.reimage was started by eevans@cumin1001 for host restbase1027.eqiad.wmnet with OS bullseye

Cookbook cookbooks.sre.hosts.reimage started by eevans@cumin1001 for host restbase1027.eqiad.wmnet with OS bullseye completed:

restbase1027 (WARN)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present
- Deleted any existing Puppet certificate
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bullseye OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202310031652_eevans_714914_restbase1027.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is not optimal, downtime not removed
- Updated Netbox data from PuppetDB