Migrate wikikube-codfw to containerd
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	JMeybohm
	Oct 22 2024, 4:53 PM

Description

This is to track the migration of all control planes and nodes of the wikikube-codfw cluster to containerd:

https://wikitech.wikimedia.org/wiki/Kubernetes/Administration/containerd_migration

Please take the chance to renumber (--move-vlan) where possible/required:
https://wikitech.wikimedia.org/wiki/Vlan_migration
To query nodes that need --move-vlan:

sudo cumin 'A:codfw and (A:wikikube-master or A:wikikube-worker) and P{F:fqdn ~ ".wmnet$"} and not A:vms and not P{F:netmask = "255.255.255.0"}'

Please take the chance to rename where possible/required:
T365571: Rename wikikube worker nodes during OS reimage

Please take the chance to deactivate hardware raid controller configs:
T358489: mw2420-mw2451 do have unnecessary raid controllers (configured)

Upcoming refreshes and expansions that will/should use bookworm and containerd right away:
T376966: wikikube-worker21[56-70] implementation tracking
T377008: wikikube-worker21[28-35] implementation tracking

Cumin query for nodes that still need reimage:
sudo cumin 'A:wikikube-worker-codfw and not P{F:lsbdistcodename = bookworm}'

Details

Related Changes in Gerrit:

Subject	Repo	Branch	Lines +/-
remove reserved name wikikube-worker2242 because of mw2282 decom	operations/puppet	production	+1 -3
Rename the remaining mw nodes to wikikube-worker224[0-2] 🥳	operations/puppet	production	+7 -17
Rename mw233[5-8] to wikikube-worker223[6-9]	operations/puppet	production	+10 -13
Rename mw235[0-3] to wikikube-worker223[2-5]	operations/puppet	production	+10 -13
Rename mw235[4-7] to wikikube-worker22[28-31]	operations/puppet	production	+11 -11
Rename mw23[59\|66\|67\|68] to wikikube-worker222[4-7]	operations/puppet	production	+11 -14
Rename mw23[69-72] to wikikube-worker222[0-3]	operations/puppet	production	+11 -18
Rename mw237[3-6] to wikikube-worker22[16-19]	operations/puppet	production	+11 -11
Rename mw241[2-5] to wikikube-worker22[12-15]	operations/puppet	production	+10 -13
Rename mw241[6-9] to wikikube-worker22[08-11]	operations/puppet	production	+11 -11
Rename kubernetes20[40-41] to wikikube-worker220[6-7]	operations/puppet	production	+5 -13
Rename kubernetes20[42-44] to wikikube-worker220[3-5]	operations/puppet	production	+8 -8
Rename kubernetes20[45-48] to wikikube-worker[2199-2202]	operations/puppet	production	+14 -10
Rename kubernetes20[49-52] to wikikube-worker219[5-8]	operations/puppet	production	+10 -10
Rename kubernetes20[53,56,58] to wikikube-worker[2192-2194]	operations/puppet	production	+8 -8
Rename kubernetes20[36-39] to wikikube-worker21[88-91]	operations/puppet	production	+9 -9
Rename kubernetes20[36-39] to wikikube-worker20(47\|66\|85\|86)	operations/puppet	production	+9 -9
Rename kubernetes20(17\|21\|22\|24) to wikikube-worker[2184-2187]	operations/puppet	production	+10 -10
Rename kubernetes[2011-2014] to wikikube-worker[2180-2183]	operations/puppet	production	+10 -10
Rename mw[2448-2451] to wikikube-worker217[6-9]	operations/puppet	production	+10 -16
Rename mw244[5-7] to wikikube-worker217[3-5]	operations/puppet	production	+9 -9
Rename mw244[02-4] to wikikube-worker201[56],wikikube-worker217[12]	operations/puppet	production	+11 -11
Rename mw243[67] to wikikube-worker200[56]	operations/puppet	production	+6 -9
preseed: Migrate wikikube-ctrl2* to containerd partitioning	operations/puppet	production	+2 -2

Related Objects
Search...

Status	Subtype	Assigned	Task
Resolved		JMeybohm	T341984 Update Kubernetes clusters to 1.31
Resolved		JMeybohm	T269684 [EPIC] Docker deprecation as a container runtime enginer for kubernetes.
Resolved		JMeybohm	T362408 Migration to containerd and away from docker
Resolved		Jelto	T377877 Migrate wikikube-codfw to containerd
Resolved		JMeybohm	T379719 wikikube-ctrl2002: Switch network cable from port 2 to port 1 on the 10G NIC
Resolved		Jhancock.wm	T381967 Relabel codfw kubernetes nodes
Resolved		Jhancock.wm	T382420 Comm Error: backplane 0 when reimaging wikikube-worker2190
Resolved		Jhancock.wm	T382422 Relabel codfw kubernetes nodes
Resolved		elukey	T383228 wikikube-worker2022 move vlan failed due to netbox timeout
Resolved	Request	Jhancock.wm	T383339 hw troubleshooting: Comm Error: backplane 0 for wikikube-worker2192.codfw.wmnet
Resolved		Jhancock.wm	T383341 Relabel codfw kubernetes nodes
Resolved		Jhancock.wm	T383595 Relabel codfw kubernetes nodes
Resolved		Jhancock.wm	T383691 Relabel codfw kubernetes nodes
Resolved		Jhancock.wm	T383764 Relabel codfw kubernetes nodes
Resolved		Jhancock.wm	T383862 Relabel codfw kubernetes nodes
Duplicate		Jelto	T383965 decommission mw2282.codfw.wmnet

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Cookbook cookbooks.sre.hosts.reimage started by jelto@cumin1002 for host wikikube-worker2228.codfw.wmnet with OS bookworm completed:

wikikube-worker2228 (PASS)
- Host successfully migrated to the new VLAN
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Add puppet_version metadata to Debian installer
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bookworm OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202501160845_jelto_3450002_wikikube-worker2228.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB

Cookbook cookbooks.sre.hosts.reimage started by jelto@cumin1002 for host wikikube-worker2229.codfw.wmnet with OS bookworm completed:

wikikube-worker2229 (PASS)
- Host successfully migrated to the new VLAN
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Add puppet_version metadata to Debian installer
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bookworm OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202501160853_jelto_3451218_wikikube-worker2229.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB

Cookbook cookbooks.sre.hosts.reimage started by jelto@cumin1002 for host wikikube-worker2230.codfw.wmnet with OS bookworm completed:

wikikube-worker2230 (PASS)
- Host successfully migrated to the new VLAN
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Add puppet_version metadata to Debian installer
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bookworm OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202501160903_jelto_3453572_wikikube-worker2230.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB

Cookbook cookbooks.sre.hosts.reimage started by jelto@cumin1002 for host wikikube-worker2231.codfw.wmnet with OS bookworm completed:

wikikube-worker2231 (PASS)
- Host successfully migrated to the new VLAN
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Add puppet_version metadata to Debian installer
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bookworm OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202501160906_jelto_3454190_wikikube-worker2231.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB

Mentioned in SAL (#wikimedia-operations) [2025-01-16T09:26:49Z] <jelto> homer 'lsw1-c6-codfw*' commit 'T377877'

Mentioned in SAL (#wikimedia-operations) [2025-01-16T09:27:46Z] <jelto> homer 'cr*codfw*' commit 'T377877'

pool host wikikube-worker[2228-2231].codfw.wmnet by jelto@cumin1002 with reason: None

Cookbook cookbooks.sre.k8s.pool-depool-node started by jelto@cumin1002 pool for host wikikube-worker[2228-2231].codfw.wmnet completed:

wikikube-worker[2228-2231].codfw.wmnet (PASS)
- Host wikikube-worker[2228-2231].codfw.wmnet pooled in wikikube-codfw

Change #1111926 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/puppet@production] Rename mw235[0-3] to wikikube-worker223[2-5]

https://gerrit.wikimedia.org/r/1111926

gerritbot added a project: Patch-For-Review.Jan 16 2025, 9:33 AM

depool host mw[2350-2353].codfw.wmnet by jelto@cumin1002 with reason: Renaming nodes

Cookbook cookbooks.sre.k8s.pool-depool-node started by jelto@cumin1002 depool for host mw[2350-2353].codfw.wmnet completed:

mw[2350-2353].codfw.wmnet (PASS)
- Host mw[2350-2353].codfw.wmnet depooled from wikikube-codfw

Change #1111926 merged by Jelto:

[operations/puppet@production] Rename mw235[0-3] to wikikube-worker223[2-5]

https://gerrit.wikimedia.org/r/1111926

Cookbook cookbooks.sre.hosts.rename started by jelto@cumin1002 from mw2350 to wikikube-worker2232 completed:

mw2350 (PASS)
- ✔️ Downtimed host on Icinga/Alertmanager
- ✔️ Disabled puppet and its timer
- ✔️ Disabled debmonitor-client timer
- ✔️ Netbox updated
- ✔️ BMC Hostname updated
- ✔️ DNS updated
- ✔️ Switch description updated
- ✔️ Removed from DebMonitor
- ✔️ Removed from Puppet master and PuppetDB
- Rename completed 👍 - now please run the re-image cookbook on the new name with --new

Cookbook cookbooks.sre.hosts.rename started by jelto@cumin1002 from mw2351 to wikikube-worker2233 completed:

mw2351 (PASS)
- ✔️ Downtimed host on Icinga/Alertmanager
- ✔️ Disabled puppet and its timer
- ✔️ Disabled debmonitor-client timer
- ✔️ Netbox updated
- ✔️ BMC Hostname updated
- ✔️ DNS updated
- ✔️ Switch description updated
- ✔️ Removed from DebMonitor
- ✔️ Removed from Puppet master and PuppetDB
- Rename completed 👍 - now please run the re-image cookbook on the new name with --new

Cookbook cookbooks.sre.hosts.rename started by jelto@cumin1002 from mw2352 to wikikube-worker2234 completed:

mw2352 (PASS)
- ✔️ Downtimed host on Icinga/Alertmanager
- ✔️ Disabled puppet and its timer
- ✔️ Disabled debmonitor-client timer
- ✔️ Netbox updated
- ✔️ BMC Hostname updated
- ✔️ DNS updated
- ✔️ Switch description updated
- ✔️ Removed from DebMonitor
- ✔️ Removed from Puppet master and PuppetDB
- Rename completed 👍 - now please run the re-image cookbook on the new name with --new

Cookbook cookbooks.sre.hosts.rename started by jelto@cumin1002 from mw2353 to wikikube-worker2235 completed:

mw2353 (WARN)
- ✔️ Downtimed host on Icinga/Alertmanager
- ✔️ Disabled puppet and its timer
- ✔️ Disabled debmonitor-client timer
- ⚠️Rollback initiated but nothing to rollback (too soon or too late).⚠️

Cookbook cookbooks.sre.hosts.rename started by jelto@cumin1002 from mw2353 to wikikube-worker2235 completed:

mw2353 (WARN)
- ✔️ Downtimed host on Icinga/Alertmanager
- ✔️ Disabled puppet and its timer
- ✔️ Disabled debmonitor-client timer
- ⚠️Rollback initiated but nothing to rollback (too soon or too late).⚠️
- ✔️ Netbox rolled back
- ⚠️Renaming failed but rollback succedded⚠️ Please check the logs for the reason and follow up with I/F if needed.Neither puppet nor alerting were re-enabled.

Cookbook cookbooks.sre.hosts.rename started by jelto@cumin1002 from mw2353 to wikikube-worker2235 completed:

mw2353 (PASS)
- ✔️ Downtimed host on Icinga/Alertmanager
- ✔️ Disabled puppet and its timer
- ✔️ Disabled debmonitor-client timer
- ✔️ Netbox updated
- ✔️ BMC Hostname updated
- ✔️ DNS updated
- ✔️ Switch description updated
- ✔️ Removed from DebMonitor
- ✔️ Removed from Puppet master and PuppetDB
- Rename completed 👍 - now please run the re-image cookbook on the new name with --new

Cookbook cookbooks.sre.k8s.renumber-node was started by jelto@cumin1002 Renumbering for host wikikube-worker2232.codfw.wmnet

Maintenance_bot removed a project: Patch-For-Review.Jan 16 2025, 10:30 AM

Cookbook cookbooks.sre.hosts.reimage was started by jelto@cumin1002 for host wikikube-worker2232.codfw.wmnet with OS bullseye

Cookbook cookbooks.sre.k8s.renumber-node was started by jelto@cumin1002 Renumbering for host wikikube-worker2233.codfw.wmnet

Cookbook cookbooks.sre.hosts.reimage was started by jelto@cumin1002 for host wikikube-worker2233.codfw.wmnet with OS bullseye

Cookbook cookbooks.sre.k8s.renumber-node was started by jelto@cumin1002 Renumbering for host wikikube-worker2234.codfw.wmnet

Cookbook cookbooks.sre.hosts.reimage was started by jelto@cumin1002 for host wikikube-worker2234.codfw.wmnet with OS bullseye

Cookbook cookbooks.sre.k8s.renumber-node was started by jelto@cumin1002 Renumbering for host wikikube-worker2235.codfw.wmnet

Cookbook cookbooks.sre.hosts.reimage was started by jelto@cumin1002 for host wikikube-worker2235.codfw.wmnet with OS bullseye

Cookbook cookbooks.sre.hosts.reimage started by jelto@cumin1002 for host wikikube-worker2232.codfw.wmnet with OS bullseye completed:

wikikube-worker2232.codfw.wmnet (PASS)

wikikube-worker2232 (PASS)
- Host successfully migrated to the new VLAN
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Add puppet_version metadata to Debian installer
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bullseye OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202501161054_jelto_3477300_wikikube-worker2232.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB

Cookbook cookbooks.sre.k8s.renumber-node started by jelto@cumin1002 Renumbering for host wikikube-worker2232.codfw.wmnet completed:

wikikube-worker2232.codfw.wmnet (FAIL)
- Successfully reimaged node wikikube-worker2232.codfw.wmnet
- Successfully set BGP to true in Netbox
- Failed to confirm homer commands

wikikube-worker2232 (PASS)
- Host successfully migrated to the new VLAN
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Add puppet_version metadata to Debian installer
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bullseye OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202501161054_jelto_3477300_wikikube-worker2232.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB

Cookbook cookbooks.sre.hosts.reimage was started by jelto@cumin1002 for host wikikube-worker2232.codfw.wmnet with OS bookworm

Cookbook cookbooks.sre.hosts.reimage started by jelto@cumin1002 for host wikikube-worker2233.codfw.wmnet with OS bullseye completed:

wikikube-worker2233.codfw.wmnet (PASS)

wikikube-worker2233 (PASS)
- Host successfully migrated to the new VLAN
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Add puppet_version metadata to Debian installer
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bullseye OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202501161101_jelto_3477906_wikikube-worker2233.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB

Cookbook cookbooks.sre.k8s.renumber-node started by jelto@cumin1002 Renumbering for host wikikube-worker2233.codfw.wmnet completed:

wikikube-worker2233.codfw.wmnet (FAIL)
- Successfully reimaged node wikikube-worker2233.codfw.wmnet
- Successfully set BGP to true in Netbox
- Failed to confirm homer commands

wikikube-worker2233 (PASS)
- Host successfully migrated to the new VLAN
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Add puppet_version metadata to Debian installer
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bullseye OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202501161101_jelto_3477906_wikikube-worker2233.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB

Cookbook cookbooks.sre.hosts.reimage was started by jelto@cumin1002 for host wikikube-worker2233.codfw.wmnet with OS bookworm

Cookbook cookbooks.sre.hosts.reimage started by jelto@cumin1002 for host wikikube-worker2234.codfw.wmnet with OS bullseye completed:

wikikube-worker2234.codfw.wmnet (PASS)

wikikube-worker2234 (PASS)
- Host successfully migrated to the new VLAN
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Add puppet_version metadata to Debian installer
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bullseye OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202501161110_jelto_3478540_wikikube-worker2234.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB

Cookbook cookbooks.sre.k8s.renumber-node started by jelto@cumin1002 Renumbering for host wikikube-worker2234.codfw.wmnet completed:

wikikube-worker2234.codfw.wmnet (FAIL)
- Successfully reimaged node wikikube-worker2234.codfw.wmnet
- Successfully set BGP to true in Netbox
- Failed to confirm homer commands

wikikube-worker2234 (PASS)
- Host successfully migrated to the new VLAN
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Add puppet_version metadata to Debian installer
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bullseye OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202501161110_jelto_3478540_wikikube-worker2234.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB

Cookbook cookbooks.sre.hosts.reimage was started by jelto@cumin1002 for host wikikube-worker2234.codfw.wmnet with OS bookworm

Cookbook cookbooks.sre.hosts.reimage started by jelto@cumin1002 for host wikikube-worker2235.codfw.wmnet with OS bullseye completed:

wikikube-worker2235.codfw.wmnet (PASS)

wikikube-worker2235 (PASS)
- Host successfully migrated to the new VLAN
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Add puppet_version metadata to Debian installer
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bullseye OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202501161117_jelto_3479188_wikikube-worker2235.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB

Cookbook cookbooks.sre.k8s.renumber-node started by jelto@cumin1002 Renumbering for host wikikube-worker2235.codfw.wmnet completed:

wikikube-worker2235.codfw.wmnet (FAIL)
- Successfully reimaged node wikikube-worker2235.codfw.wmnet
- Successfully set BGP to true in Netbox
- Failed to confirm homer commands

wikikube-worker2235 (PASS)
- Host successfully migrated to the new VLAN
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Add puppet_version metadata to Debian installer
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bullseye OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202501161117_jelto_3479188_wikikube-worker2235.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB

Cookbook cookbooks.sre.hosts.reimage was started by jelto@cumin1002 for host wikikube-worker2235.codfw.wmnet with OS bookworm

Cookbook cookbooks.sre.hosts.reimage started by jelto@cumin1002 for host wikikube-worker2232.codfw.wmnet with OS bookworm completed:

wikikube-worker2232 (PASS)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Host successfully migrated to the new VLAN
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Add puppet_version metadata to Debian installer
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bookworm OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202501161142_jelto_3487229_wikikube-worker2232.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB

Cookbook cookbooks.sre.hosts.reimage started by jelto@cumin1002 for host wikikube-worker2233.codfw.wmnet with OS bookworm completed:

wikikube-worker2233 (PASS)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Host successfully migrated to the new VLAN
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Add puppet_version metadata to Debian installer
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bookworm OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202501161145_jelto_3488906_wikikube-worker2233.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB

Cookbook cookbooks.sre.hosts.reimage started by jelto@cumin1002 for host wikikube-worker2234.codfw.wmnet with OS bookworm completed:

wikikube-worker2234 (PASS)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Host successfully migrated to the new VLAN
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Add puppet_version metadata to Debian installer
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bookworm OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202501161153_jelto_3491381_wikikube-worker2234.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB

Cookbook cookbooks.sre.hosts.reimage started by jelto@cumin1002 for host wikikube-worker2235.codfw.wmnet with OS bookworm completed:

wikikube-worker2235 (PASS)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Host successfully migrated to the new VLAN
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Add puppet_version metadata to Debian installer
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bookworm OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202501161202_jelto_3491993_wikikube-worker2235.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB

Mentioned in SAL (#wikimedia-operations) [2025-01-16T12:23:38Z] <jelto> homer 'lsw1-c6-codfw*' commit 'T377877'

Mentioned in SAL (#wikimedia-operations) [2025-01-16T12:24:32Z] <jelto> homer 'cr*codfw*' commit 'T377877'

pool host wikikube-worker[2232-2235].codfw.wmnet by jelto@cumin1002 with reason: None

Cookbook cookbooks.sre.k8s.pool-depool-node started by jelto@cumin1002 pool for host wikikube-worker[2232-2235].codfw.wmnet completed:

wikikube-worker[2232-2235].codfw.wmnet (PASS)
- Host wikikube-worker[2232-2235].codfw.wmnet pooled in wikikube-codfw

Change #1111965 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/puppet@production] Rename mw233[5-8] to wikikube-worker223[6-9]

https://gerrit.wikimedia.org/r/1111965

gerritbot added a project: Patch-For-Review.Jan 16 2025, 12:32 PM

Ladsgroup unsubscribed.Jan 16 2025, 12:32 PM

depool host mw[2335-2338].codfw.wmnet by jelto@cumin1002 with reason: Renaming nodes

Cookbook cookbooks.sre.k8s.pool-depool-node started by jelto@cumin1002 depool for host mw[2335-2338].codfw.wmnet completed:

mw[2335-2338].codfw.wmnet (PASS)
- Host mw[2335-2338].codfw.wmnet depooled from wikikube-codfw

Change #1111965 merged by Jelto:

[operations/puppet@production] Rename mw233[5-8] to wikikube-worker223[6-9]

https://gerrit.wikimedia.org/r/1111965

Cookbook cookbooks.sre.hosts.rename started by jelto@cumin1002 from mw2335 to wikikube-worker2236 completed:

mw2335 (PASS)
- ✔️ Downtimed host on Icinga/Alertmanager
- ✔️ Disabled puppet and its timer
- ✔️ Disabled debmonitor-client timer
- ✔️ Netbox updated
- ✔️ BMC Hostname updated
- ✔️ DNS updated
- ✔️ Switch description updated
- ✔️ Removed from DebMonitor
- ✔️ Removed from Puppet master and PuppetDB
- Rename completed 👍 - now please run the re-image cookbook on the new name with --new

Cookbook cookbooks.sre.hosts.rename started by jelto@cumin1002 from mw2336 to wikikube-worker2237 completed:

mw2336 (PASS)
- ✔️ Downtimed host on Icinga/Alertmanager
- ✔️ Disabled puppet and its timer
- ✔️ Disabled debmonitor-client timer
- ✔️ Netbox updated
- ✔️ BMC Hostname updated
- ✔️ DNS updated
- ✔️ Switch description updated
- ✔️ Removed from DebMonitor
- ✔️ Removed from Puppet master and PuppetDB
- Rename completed 👍 - now please run the re-image cookbook on the new name with --new

Maintenance_bot removed a project: Patch-For-Review.Jan 16 2025, 1:29 PM

Cookbook cookbooks.sre.hosts.rename started by jelto@cumin1002 from mw2337 to wikikube-worker2238 completed:

mw2337 (PASS)
- ✔️ Downtimed host on Icinga/Alertmanager
- ✔️ Disabled puppet and its timer
- ✔️ Disabled debmonitor-client timer
- ✔️ Netbox updated
- ✔️ BMC Hostname updated
- ✔️ DNS updated
- ✔️ Switch description updated
- ✔️ Removed from DebMonitor
- ✔️ Removed from Puppet master and PuppetDB
- Rename completed 👍 - now please run the re-image cookbook on the new name with --new

Cookbook cookbooks.sre.hosts.rename started by jelto@cumin1002 from mw2338 to wikikube-worker2239 completed:

mw2338 (PASS)
- ✔️ Downtimed host on Icinga/Alertmanager
- ✔️ Disabled puppet and its timer
- ✔️ Disabled debmonitor-client timer
- ✔️ Netbox updated
- ✔️ BMC Hostname updated
- ✔️ DNS updated
- ✔️ Switch description updated
- ✔️ Removed from DebMonitor
- ✔️ Removed from Puppet master and PuppetDB
- Rename completed 👍 - now please run the re-image cookbook on the new name with --new

Cookbook cookbooks.sre.hosts.reimage was started by jelto@cumin1002 for host wikikube-worker2236.codfw.wmnet with OS bookworm

Cookbook cookbooks.sre.hosts.reimage was started by jelto@cumin1002 for host wikikube-worker2237.codfw.wmnet with OS bookworm

Cookbook cookbooks.sre.hosts.reimage was started by jelto@cumin1002 for host wikikube-worker2238.codfw.wmnet with OS bookworm

Cookbook cookbooks.sre.hosts.reimage was started by jelto@cumin1002 for host wikikube-worker2239.codfw.wmnet with OS bookworm

Cookbook cookbooks.sre.hosts.reimage started by jelto@cumin1002 for host wikikube-worker2236.codfw.wmnet with OS bookworm completed:

wikikube-worker2236 (PASS)
- Host successfully migrated to the new VLAN
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Add puppet_version metadata to Debian installer
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bookworm OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202501161406_jelto_3516356_wikikube-worker2236.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB

Cookbook cookbooks.sre.hosts.reimage started by jelto@cumin1002 for host wikikube-worker2237.codfw.wmnet with OS bookworm completed:

wikikube-worker2237 (PASS)
- Host successfully migrated to the new VLAN
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Add puppet_version metadata to Debian installer
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bookworm OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202501161414_jelto_3517131_wikikube-worker2237.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB

Cookbook cookbooks.sre.hosts.reimage started by jelto@cumin1002 for host wikikube-worker2238.codfw.wmnet with OS bookworm executed with errors:

wikikube-worker2238 (FAIL)
- Host successfully migrated to the new VLAN
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console wikikube-worker2238.codfw.wmnet" to get a root shell, but depending on the failure this may not work.

Cookbook cookbooks.sre.hosts.reimage was started by jelto@cumin1002 for host wikikube-worker2238.codfw.wmnet with OS bookworm

Cookbook cookbooks.sre.hosts.reimage started by jelto@cumin1002 for host wikikube-worker2239.codfw.wmnet with OS bookworm completed:

wikikube-worker2239 (PASS)
- Host successfully migrated to the new VLAN
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Add puppet_version metadata to Debian installer
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bookworm OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202501161439_jelto_3523349_wikikube-worker2239.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB

Cookbook cookbooks.sre.hosts.reimage started by jelto@cumin1002 for host wikikube-worker2238.codfw.wmnet with OS bookworm completed:

wikikube-worker2238 (PASS)
- Host successfully migrated to the new VLAN
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Add puppet_version metadata to Debian installer
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bookworm OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202501161509_jelto_3535576_wikikube-worker2238.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB

Mentioned in SAL (#wikimedia-operations) [2025-01-16T15:31:13Z] <jelto> homer 'lsw1-c3-codfw*' commit 'T377877'

Mentioned in SAL (#wikimedia-operations) [2025-01-16T15:32:07Z] <jelto> homer 'cr*codfw*' commit 'T377877'

pool host wikikube-worker[2236-2239].codfw.wmnet by jelto@cumin1002 with reason: None

Cookbook cookbooks.sre.k8s.pool-depool-node started by jelto@cumin1002 pool for host wikikube-worker[2236-2239].codfw.wmnet completed:

wikikube-worker[2236-2239].codfw.wmnet (PASS)
- Host wikikube-worker[2236-2239].codfw.wmnet pooled in wikikube-codfw

Change #1112055 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/puppet@production] Rename the remaining mw nodes to wikikube-worker224[0-2] 🥳

https://gerrit.wikimedia.org/r/1112055

gerritbot added a project: Patch-For-Review.Jan 16 2025, 3:39 PM

depool host mw[2282,2310-2311].codfw.wmnet by jelto@cumin1002 with reason: Renaming nodes

Cookbook cookbooks.sre.k8s.pool-depool-node started by jelto@cumin1002 depool for host mw[2282,2310-2311].codfw.wmnet completed:

mw[2282,2310-2311].codfw.wmnet (PASS)
- Host mw[2282,2310-2311].codfw.wmnet depooled from wikikube-codfw

Change #1112055 merged by Jelto:

[operations/puppet@production] Rename the remaining mw nodes to wikikube-worker224[0-2] 🥳

https://gerrit.wikimedia.org/r/1112055

Cookbook cookbooks.sre.hosts.rename started by jelto@cumin1002 from mw2310 to wikikube-worker2240 completed:

mw2310 (PASS)
- ✔️ Downtimed host on Icinga/Alertmanager
- ✔️ Disabled puppet and its timer
- ✔️ Disabled debmonitor-client timer
- ✔️ Netbox updated
- ✔️ BMC Hostname updated
- ✔️ DNS updated
- ✔️ Switch description updated
- ✔️ Removed from DebMonitor
- ✔️ Removed from Puppet master and PuppetDB
- Rename completed 👍 - now please run the re-image cookbook on the new name with --new

Cookbook cookbooks.sre.hosts.rename started by jelto@cumin1002 from mw2311 to wikikube-worker2241 completed:

mw2311 (PASS)
- ✔️ Downtimed host on Icinga/Alertmanager
- ✔️ Disabled puppet and its timer
- ✔️ Disabled debmonitor-client timer
- ✔️ Netbox updated
- ✔️ BMC Hostname updated
- ✔️ DNS updated
- ✔️ Switch description updated
- ✔️ Removed from DebMonitor
- ✔️ Removed from Puppet master and PuppetDB
- Rename completed 👍 - now please run the re-image cookbook on the new name with --new

Jelto mentioned this in T383965: decommission mw2282.codfw.wmnet.Jan 17 2025, 9:20 AM

Change #1112173 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/puppet@production] remove reserved name wikikube-worker2242 because of mw2282 decom

https://gerrit.wikimedia.org/r/1112173

Change #1112173 merged by Jelto:

[operations/puppet@production] remove reserved name wikikube-worker2242 because of mw2282 decom

https://gerrit.wikimedia.org/r/1112173

Cookbook cookbooks.sre.hosts.reimage was started by jelto@cumin1002 for host wikikube-worker2240.codfw.wmnet with OS bookworm

Cookbook cookbooks.sre.hosts.reimage was started by jelto@cumin1002 for host wikikube-worker2241.codfw.wmnet with OS bookworm

Cookbook cookbooks.sre.hosts.reimage started by jelto@cumin1002 for host wikikube-worker2241.codfw.wmnet with OS bookworm executed with errors:

wikikube-worker2241 (FAIL)
- Host successfully migrated to the new VLAN
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console wikikube-worker2241.codfw.wmnet" to get a root shell, but depending on the failure this may not work.

Cookbook cookbooks.sre.hosts.reimage was started by jelto@cumin1002 for host wikikube-worker2241.codfw.wmnet with OS bookworm

Cookbook cookbooks.sre.hosts.reimage started by jelto@cumin1002 for host wikikube-worker2240.codfw.wmnet with OS bookworm completed:

wikikube-worker2240 (PASS)
- Host successfully migrated to the new VLAN
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Add puppet_version metadata to Debian installer
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bookworm OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202501170959_jelto_3701258_wikikube-worker2240.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB

Cookbook cookbooks.sre.hosts.reimage started by jelto@cumin1002 for host wikikube-worker2241.codfw.wmnet with OS bookworm completed:

wikikube-worker2241 (PASS)
- Host successfully migrated to the new VLAN
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Add puppet_version metadata to Debian installer
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bookworm OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202501171031_jelto_3705790_wikikube-worker2241.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB

Mentioned in SAL (#wikimedia-operations) [2025-01-17T10:54:24Z] <jelto> homer 'lsw1-b3-codfw*' commit 'T377877'

Mentioned in SAL (#wikimedia-operations) [2025-01-17T10:55:31Z] <jelto> homer 'cr*codfw*' commit 'T377877'

pool host wikikube-worker[2240-2241].codfw.wmnet by jelto@cumin1002 with reason: None

Cookbook cookbooks.sre.k8s.pool-depool-node started by jelto@cumin1002 pool for host wikikube-worker[2240-2241].codfw.wmnet completed:

wikikube-worker[2240-2241].codfw.wmnet (PASS)
- Host wikikube-worker[2240-2241].codfw.wmnet pooled in wikikube-codfw

All wikikube-worker in codfw are on bookworm, containerd and in the new VLANs! Thanks to @JMeybohm, @kamila and @Clement_Goubert for the support!

A short summary (just from the SAL/Phab logs):

~236 nodes have been reimaged (at least tracked in this task)
37 reimages failed, 11 other FAILS happened due to wrong cookbook usage or netbox timeouts
2 hosts had hardware issues during the reimage, 1 host was too old (T383965)

Maintenance_bot removed a project: Patch-For-Review.Jan 17 2025, 11:30 AM

Jhancock.wm closed subtask T383862: Relabel codfw kubernetes nodes as Resolved.Jan 24 2025, 5:38 PM

Migrate wikikube-codfw to containerdClosed, ResolvedPublicActions

Description

Details

Related ObjectsSearch...

Event Timeline

Migrate wikikube-codfw to containerd
Closed, ResolvedPublic
Actions

Related Objects
Search...