Rename wikikube worker nodes during OS reimage
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	JMeybohm
	May 22 2024, 9:57 AM

Description

In the process of the next OS reimage during the next k8s upgrade we should rename the wikikube worker nodes to wikikube-workerXXXX using the reimage cookbook (https://gerrit.wikimedia.org/r/c/operations/cookbooks/+/1008818) or something like a rolling-reimage cookbook for k8s.

Procedure (from T372878)

Workers

Rename

Locally, use add_k8s_node.py --puppet-dir $PUPPET_DIR --netbox-url https://netbox.wikimedia.org --netbox-token $NETBOX_TOKEN --task-id T365571 --rename-only $FQDNS (from serviceops kitchensink, make sure to pull main)
Follow the instructions
Make sure that the old names were correctly deactivated from puppetserver sudo puppet node deactivate $old_fqdn

Note: When creating a relabel task, look if there isn't one already open and edit it to cut down on the number of small tasks for DC-Ops

Details

Related Changes in Gerrit:

Subject	Repo	Branch	Lines +/-
wikikube: rename parse10[18-24] to wikikube-worker*	operations/puppet	production	+15 -26
wikikube: rename parse101[3-7] to wikikube-worker*	operations/puppet	production	+13 -16
wikikube: rename parse10[07-12] to wikikube-worker*	operations/puppet	production	+15 -15
wikikube: rename parse100[1-6] to wikikube-worker*	operations/puppet	production	+15 -15
wikikube: rename mw14[82-88] to wikikube-worker*	operations/puppet	production	+15 -22
wikikube: rename mw14[76-81] to wikikube-worker*	operations/puppet	production	+16 -19
wikikube: rename mw147[0-5] -> wikikube-worker*	operations/puppet	production	+15 -15
wikikube: rename mw146[4-9] -> wikikube-worker*	operations/puppet	production	+14 -17
wikikube: rename mw14[48-49,60-63] -> wikikube-worker101[1-6]	operations/puppet	production	+15 -18
kubernetes: rename mw14[39-42] -> wikikube-worker11[07-10]	operations/puppet	production	+11 -14
kubernetes: fix my previous host rename CR	operations/puppet	production	+1 -1
kubernetes: rename mw145[7-9] to wikikube-worker*	operations/puppet	production	+7 -10
kubernetes: rename mw145[1-5] to wikikube-worker*	operations/puppet	production	+13 -13
mw1358: Put back insetup::serviceops	operations/puppet	production	+8 -2
Rename kubernetes2032 to wikikube-worker2002	operations/puppet	production	+3 -3
Rename kubernetes2023 to wikikube-worker2001	operations/puppet	production	+6 -3
Add wikikube-worker config	operations/puppet	production	+6 -4

Related Objects
Search...

Status	Subtype	Assigned	Task
Resolved		JMeybohm	T341984 Update Kubernetes clusters to 1.31
Open		None	T336861 Fix naming confusion around main/wikikube kubernetes clusters
Resolved		Raine	T365571 Rename wikikube worker nodes during OS reimage
Duplicate		None	T366085 Relabel kubernetes2032 to wikikube-worker2002
Duplicate		None	T366468 Relabel kubernetes2023 to wikikube-worker2001
Resolved	Request	Jclark-ctr	T366583 hw troubleshooting: firmware upgrade for mw1358.eqiad.wmnet
Resolved		VRiley-WMF	T367285 Relabel eqiad wikikube worker nodes
Declined		None	T367286 Relabel codfw wikikube worker nodes
Resolved		Jhancock.wm	T381478 Relabel codfw kubernetes nodes
Resolved		Jhancock.wm	T381568 Relabel codfw kubernetes nodes

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host wikikube-worker1144.eqiad.wmnet with OS bookworm completed:

wikikube-worker1144 (PASS)
- Host successfully migrated to the new VLAN
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Add puppet_version metadata to Debian installer
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bookworm OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202501231446_kamila_1545697_wikikube-worker1144.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB

Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host wikikube-worker1145.eqiad.wmnet with OS bookworm completed:

wikikube-worker1145 (PASS)
- Host successfully migrated to the new VLAN
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Add puppet_version metadata to Debian installer
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bookworm OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202501231450_kamila_1546359_wikikube-worker1145.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB

Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host wikikube-worker1146.eqiad.wmnet with OS bookworm completed:

wikikube-worker1146 (PASS)
- Host successfully migrated to the new VLAN
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Add puppet_version metadata to Debian installer
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bookworm OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202501231453_kamila_1546614_wikikube-worker1146.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB

Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host wikikube-worker1143.eqiad.wmnet with OS bookworm completed:

wikikube-worker1143 (PASS)
- Host successfully migrated to the new VLAN
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Add puppet_version metadata to Debian installer
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bookworm OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202501231456_kamila_1545450_wikikube-worker1143.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB

Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host wikikube-worker1147.eqiad.wmnet with OS bookworm completed:

wikikube-worker1147 (PASS)
- Host successfully migrated to the new VLAN
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Add puppet_version metadata to Debian installer
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bookworm OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202501231459_kamila_1546987_wikikube-worker1147.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB

Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host wikikube-worker1142.eqiad.wmnet with OS bookworm completed:

wikikube-worker1142 (PASS)
- Host successfully migrated to the new VLAN
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Add puppet_version metadata to Debian installer
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bookworm OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202501231504_kamila_1545093_wikikube-worker1142.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB

pool host wikikube-worker[1142-1147].eqiad.wmnet by kamila@cumin1002 with reason: None

Cookbook cookbooks.sre.k8s.pool-depool-node started by kamila@cumin1002 pool for host wikikube-worker[1142-1147].eqiad.wmnet completed:

wikikube-worker[1142-1147].eqiad.wmnet (PASS)
- Host wikikube-worker[1142-1147].eqiad.wmnet pooled in wikikube-eqiad

Change #1113974 had a related patch set uploaded (by Kamila Součková; author: Kamila Součková):

[operations/puppet@production] wikikube: rename parse10[07-12] to wikikube-worker*

https://gerrit.wikimedia.org/r/1113974

gerritbot added a project: Patch-For-Review.Jan 24 2025, 11:45 AM

depool host parse[1007-1012].eqiad.wmnet by kamila@cumin1002 with reason: Renaming nodes

Change #1113974 merged by Kamila Součková:

[operations/puppet@production] wikikube: rename parse10[07-12] to wikikube-worker*

https://gerrit.wikimedia.org/r/1113974

Cookbook cookbooks.sre.k8s.pool-depool-node started by kamila@cumin1002 depool for host parse[1007-1012].eqiad.wmnet completed:

parse[1007-1012].eqiad.wmnet (PASS)
- Host parse[1007-1012].eqiad.wmnet depooled from wikikube-eqiad

Cookbook cookbooks.sre.hosts.rename started by kamila@cumin1002 from parse1007 to wikikube-worker1148 completed:

parse1007 (PASS)
- ✔️ Downtimed host on Icinga/Alertmanager
- ✔️ Disabled puppet and its timer
- ✔️ Disabled debmonitor-client timer
- ✔️ Netbox updated
- ✔️ BMC Hostname updated
- ✔️ DNS updated
- ✔️ Switch description updated
- ✔️ Removed from DebMonitor
- ✔️ Removed from Puppet master and PuppetDB
- Rename completed 👍 - now please run the re-image cookbook on the new name with --new

Cookbook cookbooks.sre.hosts.rename started by kamila@cumin1002 from parse1008 to wikikube-worker1149 completed:

parse1008 (PASS)
- ✔️ Downtimed host on Icinga/Alertmanager
- ✔️ Disabled puppet and its timer
- ✔️ Disabled debmonitor-client timer
- ✔️ Netbox updated
- ✔️ BMC Hostname updated
- ✔️ DNS updated
- ✔️ Switch description updated
- ✔️ Removed from DebMonitor
- ✔️ Removed from Puppet master and PuppetDB
- Rename completed 👍 - now please run the re-image cookbook on the new name with --new

Maintenance_bot removed a project: Patch-For-Review.Jan 24 2025, 1:30 PM

Cookbook cookbooks.sre.hosts.rename started by kamila@cumin1002 from parse1009 to wikikube-worker1150 completed:

parse1009 (PASS)
- ✔️ Downtimed host on Icinga/Alertmanager
- ✔️ Disabled puppet and its timer
- ✔️ Disabled debmonitor-client timer
- ✔️ Netbox updated
- ✔️ BMC Hostname updated
- ✔️ DNS updated
- ✔️ Switch description updated
- ✔️ Removed from DebMonitor
- ✔️ Removed from Puppet master and PuppetDB
- Rename completed 👍 - now please run the re-image cookbook on the new name with --new

Cookbook cookbooks.sre.hosts.rename started by kamila@cumin1002 from parse1010 to wikikube-worker1151 completed:

parse1010 (PASS)
- ✔️ Downtimed host on Icinga/Alertmanager
- ✔️ Disabled puppet and its timer
- ✔️ Disabled debmonitor-client timer
- ✔️ Netbox updated
- ✔️ BMC Hostname updated
- ✔️ DNS updated
- ✔️ Switch description updated
- ✔️ Removed from DebMonitor
- ✔️ Removed from Puppet master and PuppetDB
- Rename completed 👍 - now please run the re-image cookbook on the new name with --new

Cookbook cookbooks.sre.hosts.rename started by kamila@cumin1002 from parse1011 to wikikube-worker1152 completed:

parse1011 (PASS)
- ✔️ Downtimed host on Icinga/Alertmanager
- ✔️ Disabled puppet and its timer
- ✔️ Disabled debmonitor-client timer
- ✔️ Netbox updated
- ✔️ BMC Hostname updated
- ✔️ DNS updated
- ✔️ Switch description updated
- ✔️ Removed from DebMonitor
- ✔️ Removed from Puppet master and PuppetDB
- Rename completed 👍 - now please run the re-image cookbook on the new name with --new

Cookbook cookbooks.sre.hosts.rename started by kamila@cumin1002 from parse1012 to wikikube-worker1153 completed:

parse1012 (PASS)
- ✔️ Downtimed host on Icinga/Alertmanager
- ✔️ Disabled puppet and its timer
- ✔️ Disabled debmonitor-client timer
- ✔️ Netbox updated
- ✔️ BMC Hostname updated
- ✔️ DNS updated
- ✔️ Switch description updated
- ✔️ Removed from DebMonitor
- ✔️ Removed from Puppet master and PuppetDB
- Rename completed 👍 - now please run the re-image cookbook on the new name with --new

Cookbook cookbooks.sre.hosts.reimage was started by kamila@cumin1002 for host wikikube-worker1148.eqiad.wmnet with OS bookworm

Cookbook cookbooks.sre.hosts.reimage was started by kamila@cumin1002 for host wikikube-worker1149.eqiad.wmnet with OS bookworm

Cookbook cookbooks.sre.hosts.reimage was started by kamila@cumin1002 for host wikikube-worker1150.eqiad.wmnet with OS bookworm

Cookbook cookbooks.sre.hosts.reimage was started by kamila@cumin1002 for host wikikube-worker1151.eqiad.wmnet with OS bookworm

Cookbook cookbooks.sre.hosts.reimage was started by kamila@cumin1002 for host wikikube-worker1152.eqiad.wmnet with OS bookworm

Cookbook cookbooks.sre.hosts.reimage was started by kamila@cumin1002 for host wikikube-worker1153.eqiad.wmnet with OS bookworm

Change #1113989 had a related patch set uploaded (by Kamila Součková; author: Kamila Součková):

[operations/puppet@production] wikikube: rename parse101[3-7] to wikikube-worker*

https://gerrit.wikimedia.org/r/1113989

gerritbot added a project: Patch-For-Review.Jan 24 2025, 1:55 PM

Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host wikikube-worker1148.eqiad.wmnet with OS bookworm completed:

wikikube-worker1148 (PASS)
- Host successfully migrated to the new VLAN
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Add puppet_version metadata to Debian installer
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bookworm OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202501241407_kamila_1933783_wikikube-worker1148.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB

Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host wikikube-worker1149.eqiad.wmnet with OS bookworm completed:

wikikube-worker1149 (PASS)
- Host successfully migrated to the new VLAN
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Add puppet_version metadata to Debian installer
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bookworm OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202501241410_kamila_1933789_wikikube-worker1149.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB

Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host wikikube-worker1153.eqiad.wmnet with OS bookworm completed:

wikikube-worker1153 (PASS)
- Host successfully migrated to the new VLAN
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Add puppet_version metadata to Debian installer
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bookworm OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202501241413_kamila_1933867_wikikube-worker1153.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB

Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host wikikube-worker1150.eqiad.wmnet with OS bookworm completed:

wikikube-worker1150 (PASS)
- Host successfully migrated to the new VLAN
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Add puppet_version metadata to Debian installer
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bookworm OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202501241417_kamila_1933796_wikikube-worker1150.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB

Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host wikikube-worker1152.eqiad.wmnet with OS bookworm completed:

wikikube-worker1152 (PASS)
- Host successfully migrated to the new VLAN
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Add puppet_version metadata to Debian installer
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bookworm OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202501241422_kamila_1933832_wikikube-worker1152.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB

Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host wikikube-worker1151.eqiad.wmnet with OS bookworm completed:

wikikube-worker1151 (PASS)
- Host successfully migrated to the new VLAN
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Add puppet_version metadata to Debian installer
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bookworm OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202501241425_kamila_1933802_wikikube-worker1151.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB

depool host parse[1013-1017].eqiad.wmnet by kamila@cumin1002 with reason: Renaming nodes

Change #1113989 merged by Kamila Součková:

[operations/puppet@production] wikikube: rename parse101[3-7] to wikikube-worker*

https://gerrit.wikimedia.org/r/1113989

Cookbook cookbooks.sre.k8s.pool-depool-node started by kamila@cumin1002 depool for host parse[1013-1017].eqiad.wmnet completed:

parse[1013-1017].eqiad.wmnet (PASS)
- Host parse[1013-1017].eqiad.wmnet depooled from wikikube-eqiad

Cookbook cookbooks.sre.hosts.rename started by kamila@cumin1002 from parse1013 to wikikube-worker1154 completed:

parse1013 (PASS)
- ✔️ Downtimed host on Icinga/Alertmanager
- ✔️ Disabled puppet and its timer
- ✔️ Disabled debmonitor-client timer
- ✔️ Netbox updated
- ✔️ BMC Hostname updated
- ✔️ DNS updated
- ✔️ Switch description updated
- ✔️ Removed from DebMonitor
- ✔️ Removed from Puppet master and PuppetDB
- Rename completed 👍 - now please run the re-image cookbook on the new name with --new

Maintenance_bot removed a project: Patch-For-Review.Jan 24 2025, 3:30 PM

Cookbook cookbooks.sre.hosts.rename started by kamila@cumin1002 from parse1014 to wikikube-worker1155 completed:

parse1014 (PASS)
- ✔️ Downtimed host on Icinga/Alertmanager
- ✔️ Disabled puppet and its timer
- ✔️ Disabled debmonitor-client timer
- ✔️ Netbox updated
- ✔️ BMC Hostname updated
- ✔️ DNS updated
- ✔️ Switch description updated
- ✔️ Removed from DebMonitor
- ✔️ Removed from Puppet master and PuppetDB
- Rename completed 👍 - now please run the re-image cookbook on the new name with --new

Cookbook cookbooks.sre.hosts.rename started by kamila@cumin1002 from parse1015 to wikikube-worker1156 completed:

parse1015 (PASS)
- ✔️ Downtimed host on Icinga/Alertmanager
- ✔️ Disabled puppet and its timer
- ✔️ Disabled debmonitor-client timer
- ✔️ Netbox updated
- ✔️ BMC Hostname updated
- ✔️ DNS updated
- ✔️ Switch description updated
- ✔️ Removed from DebMonitor
- ✔️ Removed from Puppet master and PuppetDB
- Rename completed 👍 - now please run the re-image cookbook on the new name with --new

Cookbook cookbooks.sre.hosts.rename started by kamila@cumin1002 from parse1016 to wikikube-worker1157 completed:

parse1016 (PASS)
- ✔️ Downtimed host on Icinga/Alertmanager
- ✔️ Disabled puppet and its timer
- ✔️ Disabled debmonitor-client timer
- ✔️ Netbox updated
- ✔️ BMC Hostname updated
- ✔️ DNS updated
- ✔️ Switch description updated
- ✔️ Removed from DebMonitor
- ✔️ Removed from Puppet master and PuppetDB
- Rename completed 👍 - now please run the re-image cookbook on the new name with --new

Cookbook cookbooks.sre.hosts.rename started by kamila@cumin1002 from parse1017 to wikikube-worker1158 completed:

parse1017 (PASS)
- ✔️ Downtimed host on Icinga/Alertmanager
- ✔️ Disabled puppet and its timer
- ✔️ Disabled debmonitor-client timer
- ✔️ Netbox updated
- ✔️ BMC Hostname updated
- ✔️ DNS updated
- ✔️ Switch description updated
- ✔️ Removed from DebMonitor
- ✔️ Removed from Puppet master and PuppetDB
- Rename completed 👍 - now please run the re-image cookbook on the new name with --new

Cookbook cookbooks.sre.hosts.reimage was started by kamila@cumin1002 for host wikikube-worker1154.eqiad.wmnet with OS bookworm

Cookbook cookbooks.sre.hosts.reimage was started by kamila@cumin1002 for host wikikube-worker1155.eqiad.wmnet with OS bookworm

Cookbook cookbooks.sre.hosts.reimage was started by kamila@cumin1002 for host wikikube-worker1156.eqiad.wmnet with OS bookworm

Cookbook cookbooks.sre.hosts.reimage was started by kamila@cumin1002 for host wikikube-worker1157.eqiad.wmnet with OS bookworm

Cookbook cookbooks.sre.hosts.reimage was started by kamila@cumin1002 for host wikikube-worker1158.eqiad.wmnet with OS bookworm

Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host wikikube-worker1156.eqiad.wmnet with OS bookworm completed:

wikikube-worker1156 (PASS)
- Host successfully migrated to the new VLAN
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Add puppet_version metadata to Debian installer
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bookworm OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202501241613_kamila_1960340_wikikube-worker1156.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB

Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host wikikube-worker1158.eqiad.wmnet with OS bookworm completed:

wikikube-worker1158 (PASS)
- Host successfully migrated to the new VLAN
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Add puppet_version metadata to Debian installer
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bookworm OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202501241624_kamila_1960519_wikikube-worker1158.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB

Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host wikikube-worker1154.eqiad.wmnet with OS bookworm executed with errors:

wikikube-worker1154 (FAIL)
- Host successfully migrated to the new VLAN
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Add puppet_version metadata to Debian installer
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bookworm OS)
- Generated Puppet certificate
- The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console wikikube-worker1154.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.

Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host wikikube-worker1157.eqiad.wmnet with OS bookworm completed:

wikikube-worker1157 (PASS)
- Host successfully migrated to the new VLAN
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Add puppet_version metadata to Debian installer
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bookworm OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202501241628_kamila_1960500_wikikube-worker1157.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB

Cookbook cookbooks.sre.hosts.reimage was started by kamila@cumin1002 for host wikikube-worker1154.eqiad.wmnet with OS bookworm

Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host wikikube-worker1155.eqiad.wmnet with OS bookworm completed:

wikikube-worker1155 (PASS)
- Host successfully migrated to the new VLAN
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Add puppet_version metadata to Debian installer
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bookworm OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202501241609_kamila_1960239_wikikube-worker1155.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB

Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host wikikube-worker1154.eqiad.wmnet with OS bookworm completed:

wikikube-worker1154 (PASS)
- Host successfully migrated to the new VLAN
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Add puppet_version metadata to Debian installer
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bookworm OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202501241713_kamila_1976433_wikikube-worker1154.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB

Change #1114354 had a related patch set uploaded (by Kamila Součková; author: Kamila Součková):

[operations/puppet@production] wikikube: rename parse10[18-24] to wikikube-worker*

https://gerrit.wikimedia.org/r/1114354

gerritbot added a project: Patch-For-Review.Jan 27 2025, 11:46 AM

depool host parse[1018-1024].eqiad.wmnet by kamila@cumin1002 with reason: Renaming nodes

Change #1114354 merged by Kamila Součková:

[operations/puppet@production] wikikube: rename parse10[18-24] to wikikube-worker*

https://gerrit.wikimedia.org/r/1114354

Cookbook cookbooks.sre.k8s.pool-depool-node started by kamila@cumin1002 depool for host parse[1018-1024].eqiad.wmnet completed:

parse[1018-1024].eqiad.wmnet (PASS)
- Host parse[1018-1024].eqiad.wmnet depooled from wikikube-eqiad

Cookbook cookbooks.sre.hosts.rename started by kamila@cumin1002 from parse1018 to wikikube-worker1159 completed:

parse1018 (PASS)
- ✔️ Downtimed host on Icinga/Alertmanager
- ✔️ Disabled puppet and its timer
- ✔️ Disabled debmonitor-client timer
- ✔️ Netbox updated
- ✔️ BMC Hostname updated
- ✔️ DNS updated
- ✔️ Switch description updated
- ✔️ Removed from DebMonitor
- ✔️ Removed from Puppet master and PuppetDB
- Rename completed 👍 - now please run the re-image cookbook on the new name with --new

Cookbook cookbooks.sre.hosts.rename started by kamila@cumin1002 from parse1019 to wikikube-worker1160 completed:

parse1019 (PASS)
- ✔️ Downtimed host on Icinga/Alertmanager
- ✔️ Disabled puppet and its timer
- ✔️ Disabled debmonitor-client timer
- ✔️ Netbox updated
- ✔️ BMC Hostname updated
- ✔️ DNS updated
- ✔️ Switch description updated
- ✔️ Removed from DebMonitor
- ✔️ Removed from Puppet master and PuppetDB
- Rename completed 👍 - now please run the re-image cookbook on the new name with --new

Cookbook cookbooks.sre.hosts.rename started by kamila@cumin1002 from parse1020 to wikikube-worker1161 completed:

parse1020 (PASS)
- ✔️ Downtimed host on Icinga/Alertmanager
- ✔️ Disabled puppet and its timer
- ✔️ Disabled debmonitor-client timer
- ✔️ Netbox updated
- ✔️ BMC Hostname updated
- ✔️ DNS updated
- ✔️ Switch description updated
- ✔️ Removed from DebMonitor
- ✔️ Removed from Puppet master and PuppetDB
- Rename completed 👍 - now please run the re-image cookbook on the new name with --new

Cookbook cookbooks.sre.hosts.rename started by kamila@cumin1002 from parse1021 to wikikube-worker1162 completed:

parse1021 (PASS)
- ✔️ Downtimed host on Icinga/Alertmanager
- ✔️ Disabled puppet and its timer
- ✔️ Disabled debmonitor-client timer
- ✔️ Netbox updated
- ✔️ BMC Hostname updated
- ✔️ DNS updated
- ✔️ Switch description updated
- ✔️ Removed from DebMonitor
- ✔️ Removed from Puppet master and PuppetDB
- Rename completed 👍 - now please run the re-image cookbook on the new name with --new

Cookbook cookbooks.sre.hosts.rename started by kamila@cumin1002 from parse1022 to wikikube-worker1163 completed:

parse1022 (PASS)
- ✔️ Downtimed host on Icinga/Alertmanager
- ✔️ Disabled puppet and its timer
- ✔️ Disabled debmonitor-client timer
- ✔️ Netbox updated
- ✔️ BMC Hostname updated
- ✔️ DNS updated
- ✔️ Switch description updated
- ✔️ Removed from DebMonitor
- ✔️ Removed from Puppet master and PuppetDB
- Rename completed 👍 - now please run the re-image cookbook on the new name with --new

Cookbook cookbooks.sre.hosts.rename started by kamila@cumin1002 from parse1023 to wikikube-worker1164 completed:

parse1023 (PASS)
- ✔️ Downtimed host on Icinga/Alertmanager
- ✔️ Disabled puppet and its timer
- ✔️ Disabled debmonitor-client timer
- ✔️ Netbox updated
- ✔️ BMC Hostname updated
- ✔️ DNS updated
- ✔️ Switch description updated
- ✔️ Removed from DebMonitor
- ✔️ Removed from Puppet master and PuppetDB
- Rename completed 👍 - now please run the re-image cookbook on the new name with --new

Cookbook cookbooks.sre.hosts.rename started by kamila@cumin1002 from parse1024 to wikikube-worker1165 completed:

parse1024 (PASS)
- ✔️ Downtimed host on Icinga/Alertmanager
- ✔️ Disabled puppet and its timer
- ✔️ Disabled debmonitor-client timer
- ✔️ Netbox updated
- ✔️ BMC Hostname updated
- ✔️ DNS updated
- ✔️ Switch description updated
- ✔️ Removed from DebMonitor
- ✔️ Removed from Puppet master and PuppetDB
- Rename completed 👍 - now please run the re-image cookbook on the new name with --new

Cookbook cookbooks.sre.hosts.reimage was started by kamila@cumin1002 for host wikikube-worker1160.eqiad.wmnet with OS bookworm

Cookbook cookbooks.sre.hosts.reimage was started by kamila@cumin1002 for host wikikube-worker1161.eqiad.wmnet with OS bookworm

Cookbook cookbooks.sre.hosts.reimage was started by kamila@cumin1002 for host wikikube-worker1162.eqiad.wmnet with OS bookworm

Cookbook cookbooks.sre.hosts.reimage was started by kamila@cumin1002 for host wikikube-worker1163.eqiad.wmnet with OS bookworm

Cookbook cookbooks.sre.hosts.reimage was started by kamila@cumin1002 for host wikikube-worker1164.eqiad.wmnet with OS bookworm

Cookbook cookbooks.sre.hosts.reimage was started by kamila@cumin1002 for host wikikube-worker1165.eqiad.wmnet with OS bookworm

Cookbook cookbooks.sre.hosts.reimage was started by kamila@cumin1002 for host wikikube-worker1159.eqiad.wmnet with OS bookworm

Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host wikikube-worker1160.eqiad.wmnet with OS bookworm completed:

wikikube-worker1160 (PASS)
- Host successfully migrated to the new VLAN
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Add puppet_version metadata to Debian installer
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bookworm OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202501271532_kamila_2603493_wikikube-worker1160.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB

Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host wikikube-worker1161.eqiad.wmnet with OS bookworm completed:

wikikube-worker1161 (PASS)
- Host successfully migrated to the new VLAN
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Add puppet_version metadata to Debian installer
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bookworm OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202501271536_kamila_2603523_wikikube-worker1161.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB

Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host wikikube-worker1162.eqiad.wmnet with OS bookworm completed:

wikikube-worker1162 (PASS)
- Host successfully migrated to the new VLAN
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Add puppet_version metadata to Debian installer
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bookworm OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202501271539_kamila_2603751_wikikube-worker1162.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB

Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host wikikube-worker1159.eqiad.wmnet with OS bookworm completed:

wikikube-worker1159 (PASS)
- Host successfully migrated to the new VLAN
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Add puppet_version metadata to Debian installer
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bookworm OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202501271542_kamila_2604205_wikikube-worker1159.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB

Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host wikikube-worker1165.eqiad.wmnet with OS bookworm completed:

wikikube-worker1165 (PASS)
- Host successfully migrated to the new VLAN
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Add puppet_version metadata to Debian installer
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bookworm OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202501271546_kamila_2603858_wikikube-worker1165.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB

Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host wikikube-worker1164.eqiad.wmnet with OS bookworm executed with errors:

wikikube-worker1164 (FAIL)
- Host successfully migrated to the new VLAN
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Add puppet_version metadata to Debian installer
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bookworm OS)
- Generated Puppet certificate
- The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console wikikube-worker1164.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.

Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host wikikube-worker1163.eqiad.wmnet with OS bookworm completed:

wikikube-worker1163 (PASS)
- Host successfully migrated to the new VLAN
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Add puppet_version metadata to Debian installer
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bookworm OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202501271549_kamila_2603763_wikikube-worker1163.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB

Cookbook cookbooks.sre.hosts.reimage was started by kamila@cumin1002 for host wikikube-worker1164.eqiad.wmnet with OS bookworm

Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host wikikube-worker1164.eqiad.wmnet with OS bookworm completed:

wikikube-worker1164 (PASS)
- Host successfully migrated to the new VLAN
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Add puppet_version metadata to Debian installer
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bookworm OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202501271628_kamila_2624913_wikikube-worker1164.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB

pool host wikikube-worker[1159-1165].eqiad.wmnet by kamila@cumin1002 with reason: None

Cookbook cookbooks.sre.k8s.pool-depool-node started by kamila@cumin1002 pool for host wikikube-worker[1159-1165].eqiad.wmnet completed:

wikikube-worker[1159-1165].eqiad.wmnet (PASS)
- Host wikikube-worker[1159-1165].eqiad.wmnet pooled in wikikube-eqiad

All done 🎉

Clement_Goubert awarded a token.Jan 28 2025, 11:44 AM

pool host wikikube-worker[1124-1128].eqiad.wmnet by kamila@cumin1002 with reason: reimage complete

Cookbook cookbooks.sre.k8s.pool-depool-node started by kamila@cumin1002 pool for host wikikube-worker[1124-1128].eqiad.wmnet completed:

wikikube-worker[1124-1128].eqiad.wmnet (PASS)
- Host wikikube-worker[1124-1128].eqiad.wmnet pooled in wikikube-eqiad

Maintenance_bot removed a project: Patch-For-Review.Feb 12 2025, 10:31 PM

pool host wikikube-worker[1148-1153].eqiad.wmnet by kamila@cumin1002 with reason: reimage complete

Cookbook cookbooks.sre.k8s.pool-depool-node started by kamila@cumin1002 pool for host wikikube-worker[1148-1153].eqiad.wmnet completed:

wikikube-worker[1148-1153].eqiad.wmnet (PASS)
- Host wikikube-worker[1148-1153].eqiad.wmnet pooled in wikikube-eqiad

pool host wikikube-worker1123.eqiad.wmnet by kamila@cumin1002 with reason: reimage complete

Cookbook cookbooks.sre.k8s.pool-depool-node started by kamila@cumin1002 pool for host wikikube-worker1123.eqiad.wmnet completed:

wikikube-worker1123.eqiad.wmnet (PASS)
- Host wikikube-worker1123.eqiad.wmnet pooled in wikikube-eqiad

Rename wikikube worker nodes during OS reimageClosed, ResolvedPublicActions