Page MenuHomePhabricator
Create Task
Maniphest T408757

Q2:rack/setup/install wikikube-worker2332-56
Closed, ResolvedPublic
Actions

Description

This task will track the racking, setup, and OS installation of wikikube-worker2332-56

Hostname / Racking / Installation Details

This section should list the racking restrictions for these hosts. If they shouldn't share a rack/row with one another, or any existing hosts. This section should also list the other details listed below.

Hostnames: What are the hostnames, and have you updated https://wikitech.wikimedia.org/wiki/Infrastructure_naming_conventions ? wikikube-worker23[32-56]
Racking Proposal: Where should these systems be racked? Can they share with any existing systems or should they avoid any other systems sharing their rack or row? Use https://fault-tolerance.toolforge.org/map to optimize this placement.

  • Row A: 4
  • Row B: 2
  • Row C: 6
  • Row D: 13

Networking Setup: # of Connections:1/2 - Speed:1G/10G. - VLAN:Private/Public/Other(Specify) :
OS Distro: Bookworm (default unless otherwise specified)
Boot Method: Legacy BIOS or UEFI. Please note UEFI must have partman updates applied in advance of setup and is currently in pilot program: https://wikitech.wikimedia.org/wiki/UEFI_Boot
Sub-team Technical Contact: Who should our on-sites contact with any questions involving system racking and setup? @Clement_Goubert

Per host setup checklist

Each host should have its own setup checklist copied and pasted into the list below.

wikikube-worker2332 ~Jenn
  • Receive in system on procurement task T405285 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the [[ https://netbox.wikimedia.org/extras/scripts/provision_server.ProvisionServerNetwork/ | Provision a server's network attributes [x] Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
wikikube-worker2333 ~Jenn
  • Receive in system on procurement task T405285 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the [[ https://netbox.wikimedia.org/extras/scripts/provision_server.ProvisionServerNetwork/ | Provision a server's network attributes [x] Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
wikikube-worker2334 ~Jenn
  • Receive in system on procurement task T405285 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the [[ https://netbox.wikimedia.org/extras/scripts/provision_server.ProvisionServerNetwork/ | Provision a server's network attributes [x] Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
wikikube-worker2335 ~Jenn
  • Receive in system on procurement task T405285 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the [[ https://netbox.wikimedia.org/extras/scripts/provision_server.ProvisionServerNetwork/ | Provision a server's network attributes [x] Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
wikikube-worker2336 ~Jenn
  • Receive in system on procurement task T405285 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the [[ https://netbox.wikimedia.org/extras/scripts/provision_server.ProvisionServerNetwork/ | Provision a server's network attributes [x] Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
wikikube-worker2337 ~Jenn
  • Receive in system on procurement task T405285 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the [[ https://netbox.wikimedia.org/extras/scripts/provision_server.ProvisionServerNetwork/ | Provision a server's network attributes [x] Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
wikikube-worker2338 ~Jenn
  • Receive in system on procurement task T405285 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the [[ https://netbox.wikimedia.org/extras/scripts/provision_server.ProvisionServerNetwork/ | Provision a server's network attributes [x] Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
wikikube-worker2339 ~Jenn
  • Receive in system on procurement task T405285 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the [[ https://netbox.wikimedia.org/extras/scripts/provision_server.ProvisionServerNetwork/ | Provision a server's network attributes [x] Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
wikikube-worker2340 ~Jenn
  • Receive in system on procurement task T405285 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the [[ https://netbox.wikimedia.org/extras/scripts/provision_server.ProvisionServerNetwork/ | Provision a server's network attributes [x] Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
wikikube-worker2341 ~Jenn
  • Receive in system on procurement task T405285 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the [[ https://netbox.wikimedia.org/extras/scripts/provision_server.ProvisionServerNetwork/ | Provision a server's network attributes [x] Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
wikikube-worker2342
  • Receive in system on procurement task T405285 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the [[ https://netbox.wikimedia.org/extras/scripts/provision_server.ProvisionServerNetwork/ | Provision a server's network attributes [x] Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
wikikube-worker2343 ~Jenn
  • Receive in system on procurement task T405285 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the [[ https://netbox.wikimedia.org/extras/scripts/provision_server.ProvisionServerNetwork/ | Provision a server's network attributes [x] Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
wikikube-worker2344 ~Jenn
  • Receive in system on procurement task T405285 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the [[ https://netbox.wikimedia.org/extras/scripts/provision_server.ProvisionServerNetwork/ | Provision a server's network attributes [x]Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
wikikube-worker2345 ~Jenn
  • Receive in system on procurement task T405285 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the [[ https://netbox.wikimedia.org/extras/scripts/provision_server.ProvisionServerNetwork/ | Provision a server's network attributes [x] Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
wikikube-worker2346

Note: this machine arrived broken. open ticket in T414708

  • Receive in system on procurement task T405285 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the [[ https://netbox.wikimedia.org/extras/scripts/provision_server.ProvisionServerNetwork/ | Provision a server's network attributes [] Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
wikikube-worker2347
  • Receive in system on procurement task T405285 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the [[ https://netbox.wikimedia.org/extras/scripts/provision_server.ProvisionServerNetwork/ | Provision a server's network attributes [x] Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
wikikube-worker2348
  • Receive in system on procurement task T405285 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the [[ https://netbox.wikimedia.org/extras/scripts/provision_server.ProvisionServerNetwork/ | Provision a server's network attributes [x] Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
wikikube-worker2349
  • Receive in system on procurement task T405285 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the [[ https://netbox.wikimedia.org/extras/scripts/provision_server.ProvisionServerNetwork/ | Provision a server's network attributes [x] Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
wikikube-worker2350
  • Receive in system on procurement task T405285 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the [[ https://netbox.wikimedia.org/extras/scripts/provision_server.ProvisionServerNetwork/ | Provision a server's network attributes [x] Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
wikikube-worker2351
  • Receive in system on procurement task T405285 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the [[ https://netbox.wikimedia.org/extras/scripts/provision_server.ProvisionServerNetwork/ | Provision a server's network attributes [x] Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
wikikube-worker2352
  • Receive in system on procurement task T405285 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the [[ https://netbox.wikimedia.org/extras/scripts/provision_server.ProvisionServerNetwork/ | Provision a server's network attributes [x] Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
wikikube-worker2353
  • Receive in system on procurement task T405285 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the [[ https://netbox.wikimedia.org/extras/scripts/provision_server.ProvisionServerNetwork/ | Provision a server's network attributes [x] Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
wikikube-worker2354
  • Receive in system on procurement task T405285 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the [[ https://netbox.wikimedia.org/extras/scripts/provision_server.ProvisionServerNetwork/ | Provision a server's network attributes [x] Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
wikikube-worker2355
  • Receive in system on procurement task T405285 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the [[ https://netbox.wikimedia.org/extras/scripts/provision_server.ProvisionServerNetwork/ | Provision a server's network attributes [x] Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
wikikube-worker2356
  • Receive in system on procurement task T405285 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the [[ https://netbox.wikimedia.org/extras/scripts/provision_server.ProvisionServerNetwork/ | Provision a server's network attributes [x] Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook

Details

Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
Add Nokia BGP routing policy for wikikube-worker / k8s hostsCathal Mooneyoperations/homer/publicmaster+84 -0
site.pp: Add wikikube-worker23[32-56]Clément Goubertoperations/puppetproduction+5 -0
partman: New wikikube-worker need UEFIClément Goubertoperations/puppetproduction+9 -0
site.pp: Add new wikikube insetup hostsClément Goubertoperations/puppetproduction+20 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Maintenance_bot removed a project: Patch-For-Review.Jan 16 2026, 4:31 PM
Jhancock.wm updated the task description. (Show Details)Jan 16 2026, 4:37 PM
Papaul updated the task description. (Show Details)Jan 16 2026, 4:41 PM
Jhancock.wm updated the task description. (Show Details)Jan 16 2026, 4:52 PM
Jhancock.wm updated the task description. (Show Details)
Papaul added a comment.Jan 16 2026, 4:57 PM

@Clement_Goubert thanks

Papaul updated the task description. (Show Details)Jan 16 2026, 5:05 PM
Papaul updated the task description. (Show Details)Jan 16 2026, 5:20 PM
Papaul updated the task description. (Show Details)Jan 16 2026, 5:36 PM
Papaul updated the task description. (Show Details)Jan 16 2026, 5:48 PM
Papaul updated the task description. (Show Details)Jan 16 2026, 5:55 PM
Papaul updated the task description. (Show Details)Jan 16 2026, 6:21 PM
Jhancock.wm updated the task description. (Show Details)Jan 16 2026, 7:07 PM
Jhancock.wm updated the task description. (Show Details)Jan 16 2026, 7:13 PM

wikikube-worker2345 is not provisioning. will check next time on site.

ops-monitoring-bot added a comment.Jan 16 2026, 7:17 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin1003 for host wikikube-worker2332.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Jan 16 2026, 7:18 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin1003 for host wikikube-worker2333.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Jan 16 2026, 7:18 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin1003 for host wikikube-worker2334.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Jan 16 2026, 7:53 PM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin1003 for host wikikube-worker2332.codfw.wmnet with OS bookworm completed:

  • wikikube-worker2332 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202601161935_jhancock_1950963_wikikube-worker2332.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Jan 16 2026, 7:55 PM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin1003 for host wikikube-worker2333.codfw.wmnet with OS bookworm completed:

  • wikikube-worker2333 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202601161938_jhancock_1950988_wikikube-worker2333.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Jan 16 2026, 8:01 PM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin1003 for host wikikube-worker2334.codfw.wmnet with OS bookworm completed:

  • wikikube-worker2334 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202601161942_jhancock_1951024_wikikube-worker2334.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
Jhancock.wm updated the task description. (Show Details)Jan 16 2026, 8:01 PM
ops-monitoring-bot added a comment.Jan 16 2026, 8:02 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin1003 for host wikikube-worker2335.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Jan 16 2026, 8:03 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin1003 for host wikikube-worker2336.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Jan 16 2026, 8:03 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin1003 for host wikikube-worker2337.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Jan 16 2026, 8:37 PM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin1003 for host wikikube-worker2336.codfw.wmnet with OS bookworm completed:

  • wikikube-worker2336 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202601162018_jhancock_1963196_wikikube-worker2336.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Jan 16 2026, 8:42 PM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin1003 for host wikikube-worker2337.codfw.wmnet with OS bookworm completed:

  • wikikube-worker2337 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202601162022_jhancock_1963234_wikikube-worker2337.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Jan 16 2026, 8:45 PM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin1003 for host wikikube-worker2335.codfw.wmnet with OS bookworm completed:

  • wikikube-worker2335 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202601162026_jhancock_1963183_wikikube-worker2335.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
Jhancock.wm updated the task description. (Show Details)Jan 16 2026, 8:46 PM
ops-monitoring-bot added a comment.Jan 16 2026, 8:48 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin1003 for host wikikube-worker2338.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Jan 16 2026, 8:48 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin1003 for host wikikube-worker2339.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Jan 16 2026, 8:49 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin1003 for host wikikube-worker2340.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Jan 16 2026, 9:20 PM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin1003 for host wikikube-worker2338.codfw.wmnet with OS bookworm completed:

  • wikikube-worker2338 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202601162103_jhancock_1973241_wikikube-worker2338.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Jan 16 2026, 9:24 PM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin1003 for host wikikube-worker2339.codfw.wmnet with OS bookworm completed:

  • wikikube-worker2339 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202601162106_jhancock_1973258_wikikube-worker2339.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Jan 16 2026, 9:32 PM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin1003 for host wikikube-worker2340.codfw.wmnet with OS bookworm completed:

  • wikikube-worker2340 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202601162113_jhancock_1973276_wikikube-worker2340.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
Jhancock.wm updated the task description. (Show Details)Jan 16 2026, 9:33 PM
ops-monitoring-bot added a comment.Jan 16 2026, 9:35 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin1003 for host wikikube-worker2341.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Jan 16 2026, 9:35 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin1003 for host wikikube-worker2342.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Jan 16 2026, 9:35 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin1003 for host wikikube-worker2343.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Jan 16 2026, 10:09 PM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin1003 for host wikikube-worker2341.codfw.wmnet with OS bookworm completed:

  • wikikube-worker2341 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202601162150_jhancock_1985616_wikikube-worker2341.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Jan 16 2026, 10:15 PM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin1003 for host wikikube-worker2342.codfw.wmnet with OS bookworm completed:

  • wikikube-worker2342 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202601162154_jhancock_1985656_wikikube-worker2342.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Jan 16 2026, 10:15 PM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin1003 for host wikikube-worker2343.codfw.wmnet with OS bookworm completed:

  • wikikube-worker2343 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202601162158_jhancock_1985674_wikikube-worker2343.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
Jhancock.wm updated the task description. (Show Details)Jan 16 2026, 10:20 PM
ops-monitoring-bot added a comment.Jan 16 2026, 10:22 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin1003 for host wikikube-worker2344.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Jan 16 2026, 10:23 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin1003 for host wikikube-worker2347.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Jan 16 2026, 10:23 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin1003 for host wikikube-worker2348.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Jan 16 2026, 10:57 PM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin1003 for host wikikube-worker2344.codfw.wmnet with OS bookworm completed:

  • wikikube-worker2344 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202601162238_jhancock_1998027_wikikube-worker2344.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Jan 16 2026, 10:59 PM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin1003 for host wikikube-worker2348.codfw.wmnet with OS bookworm completed:

  • wikikube-worker2348 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202601162242_jhancock_1998091_wikikube-worker2348.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Jan 16 2026, 11:04 PM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin1003 for host wikikube-worker2347.codfw.wmnet with OS bookworm completed:

  • wikikube-worker2347 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202601162246_jhancock_1998040_wikikube-worker2347.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
Jhancock.wm updated the task description. (Show Details)Jan 16 2026, 11:06 PM
Jhancock.wm added a subscriber: cmooney.EditedJan 20 2026, 4:20 PM

@cmooney these have had os installs from nokia switches so far.

E 2:
wikikube-worker2334
wikikube-worker2335
wikikube-worker2336

E 4:
wikikube-worker2339
wikikube-worker2340
wikikube-worker2341

E 5:
wikikube-worker2342
wikikube-worker2343
wikikube-worker2344

these are still pending in F2 and F4. should have them done before you get online tomorrow.
wikikube-worker2348 (finished now)
wikikube-worker2349
wikikube-worker2350
wikikube-worker2354
wikikube-worker2355
wikikube-worker2356

cmooney added a comment.Jan 20 2026, 5:02 PM
In T408757#11538119, @Jhancock.wm wrote:

@cmooney these have had os installs from nokia switches so far.

Thank you! Excellent work.

In T408757#11520421, @Clement_Goubert wrote:

Note for @jasmine_ (or whoever ends up putting them in production), these servers need to be excluded from the LVS backends.

Just a note from me on these that I need to prep the BGP policy for wikikube-workers on for the Nokia platform. So whenever they are ready we should pick one test host from the list in the last comment and check everything works as expected before doing the whole batch.

ops-monitoring-bot added a comment.Jan 20 2026, 7:31 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin1003 for host wikikube-worker2345.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Jan 20 2026, 7:31 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin1003 for host wikikube-worker2349.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Jan 20 2026, 7:32 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin1003 for host wikikube-worker2350.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Jan 20 2026, 8:08 PM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin1003 for host wikikube-worker2345.codfw.wmnet with OS bookworm completed:

  • wikikube-worker2345 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202601201949_jhancock_2639527_wikikube-worker2345.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Jan 20 2026, 8:12 PM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin1003 for host wikikube-worker2349.codfw.wmnet with OS bookworm completed:

  • wikikube-worker2349 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202601201954_jhancock_2639812_wikikube-worker2349.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Jan 20 2026, 8:15 PM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin1003 for host wikikube-worker2350.codfw.wmnet with OS bookworm completed:

  • wikikube-worker2350 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202601201958_jhancock_2640114_wikikube-worker2350.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
Jhancock.wm updated the task description. (Show Details)Jan 20 2026, 8:27 PM
ops-monitoring-bot added a comment.Jan 20 2026, 8:29 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin1003 for host wikikube-worker2351.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Jan 20 2026, 8:30 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin1003 for host wikikube-worker2352.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Jan 20 2026, 8:30 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin1003 for host wikikube-worker2353.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Jan 20 2026, 9:03 PM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin1003 for host wikikube-worker2351.codfw.wmnet with OS bookworm completed:

  • wikikube-worker2351 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202601202044_jhancock_2704684_wikikube-worker2351.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Jan 20 2026, 9:08 PM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin1003 for host wikikube-worker2352.codfw.wmnet with OS bookworm completed:

  • wikikube-worker2352 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202601202048_jhancock_2704933_wikikube-worker2352.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Jan 20 2026, 9:10 PM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin1003 for host wikikube-worker2353.codfw.wmnet with OS bookworm completed:

  • wikikube-worker2353 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202601202052_jhancock_2705238_wikikube-worker2353.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
gerritbot added a comment.Jan 21 2026, 1:30 PM

Change #1229562 had a related patch set uploaded (by Cathal Mooney; author: Cathal Mooney):

[operations/homer/public@master] Add Nokia BGP routing policy for wikikube-worker / k8s hosts

https://gerrit.wikimedia.org/r/1229562

gerritbot added a project: Patch-For-Review.Jan 21 2026, 1:30 PM
Jhancock.wm updated the task description. (Show Details)Jan 21 2026, 4:11 PM
ops-monitoring-bot added a comment.Jan 21 2026, 4:24 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin1003 for host wikikube-worker2354.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Jan 21 2026, 4:25 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin1003 for host wikikube-worker2355.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Jan 21 2026, 4:25 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin1003 for host wikikube-worker2356.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Jan 21 2026, 4:51 PM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin1003 for host wikikube-worker2354.codfw.wmnet with OS bookworm executed with errors:

  • wikikube-worker2354 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202601211640_jhancock_2934424_wikikube-worker2354.out
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console wikikube-worker2354.codfw.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Jan 21 2026, 4:53 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin1003 for host wikikube-worker2354.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Jan 21 2026, 5:06 PM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin1003 for host wikikube-worker2355.codfw.wmnet with OS bookworm completed:

  • wikikube-worker2355 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202601211644_jhancock_2934440_wikikube-worker2355.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Jan 21 2026, 5:45 PM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin1003 for host wikikube-worker2356.codfw.wmnet with OS bookworm executed with errors:

  • wikikube-worker2356 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console wikikube-worker2356.codfw.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Jan 21 2026, 6:09 PM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin1003 for host wikikube-worker2354.codfw.wmnet with OS bookworm completed:

  • wikikube-worker2354 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202601211709_jhancock_2939107_wikikube-worker2354.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Jan 22 2026, 4:17 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin1003 for host wikikube-worker2354.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Jan 22 2026, 4:34 PM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin1003 for host wikikube-worker2354.codfw.wmnet with OS bookworm executed with errors:

  • wikikube-worker2354 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Unable to disable Puppet, the host may have been unreachable
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run failed and logged in /var/log/spicerack/sre/hosts/reimage/202601221633_jhancock_3101715_wikikube-worker2354.out, asking the operator what to do
    • First Puppet run failed and the operator aborted
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console wikikube-worker2354.codfw.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Jan 22 2026, 4:40 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin1003 for host wikikube-worker2354.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Jan 22 2026, 5:17 PM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin1003 for host wikikube-worker2354.codfw.wmnet with OS bookworm completed:

  • wikikube-worker2354 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Unable to disable Puppet, the host may have been unreachable
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202601221659_jhancock_3104939_wikikube-worker2354.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Jan 22 2026, 5:20 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin1003 for host wikikube-worker2355.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Jan 22 2026, 6:00 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin1003 for host wikikube-worker2356.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Jan 22 2026, 6:09 PM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin1003 for host wikikube-worker2355.codfw.wmnet with OS bookworm completed:

  • wikikube-worker2355 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Unable to disable Puppet, the host may have been unreachable
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202601221749_jhancock_3110606_wikikube-worker2355.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Jan 22 2026, 6:59 PM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin1003 for host wikikube-worker2356.codfw.wmnet with OS bookworm completed:

  • wikikube-worker2356 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202601221818_jhancock_3116575_wikikube-worker2356.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
Jhancock.wm updated the task description. (Show Details)Jan 22 2026, 7:05 PM
Jhancock.wm added a comment.Jan 22 2026, 7:10 PM

@Clement_Goubert all of the servers except wikikube-worker2346 are installed and ready for you. that one was essentially dead out of the box. I opened a case with supermicro last week but haven't heard anything from them other than a "we got your message" email. separate troubleshooting ticket on T414708.

The question is, do you care if i close this ticket and install the other server once it's repaired from the other ticket? don't wanna bork a work flow.

Clement_Goubert added a comment.Jan 23 2026, 8:32 AM
In T408757#11546627, @Jhancock.wm wrote:

@Clement_Goubert all of the servers except wikikube-worker2346 are installed and ready for you. that one was essentially dead out of the box. I opened a case with supermicro last week but haven't heard anything from them other than a "we got your message" email. separate troubleshooting ticket on T414708.

The question is, do you care if i close this ticket and install the other server once it's repaired from the other ticket? don't wanna bork a work flow.

We're good, close it :) Thank you

Jhancock.wm closed this task as Resolved.Jan 27 2026, 11:37 AM
gerritbot added a comment.Feb 4 2026, 12:02 PM

Change #1229562 merged by jenkins-bot:

[operations/homer/public@master] Add Nokia BGP routing policy for wikikube-worker / k8s hosts

https://gerrit.wikimedia.org/r/1229562

Maintenance_bot removed a project: Patch-For-Review.Feb 4 2026, 12:32 PM
Clement_Goubert created subtask T417772: wikikube-worker23[32-56] implementation tracking.Feb 18 2026, 12:11 PM
Clement_Goubert changed the status of subtask T417772: wikikube-worker23[32-56] implementation tracking from Open to In Progress.Feb 18 2026, 3:13 PM
Clement_Goubert changed the status of subtask T417772: wikikube-worker23[32-56] implementation tracking from In Progress to Stalled.Feb 19 2026, 2:43 PM
Raine changed the status of subtask T417772: wikikube-worker23[32-56] implementation tracking from Stalled to Open.Feb 25 2026, 2:16 PM
MLechvien-WMF mentioned this in T418212: Automate the creation of implementation task from rack/setup/install tasks for Serviceops.Feb 26 2026, 9:20 AM
Clement_Goubert changed the status of subtask T417772: wikikube-worker23[32-56] implementation tracking from Open to In Progress.Mar 2 2026, 3:06 PM
Blake closed subtask T417772: wikikube-worker23[32-56] implementation tracking as Resolved.Mar 6 2026, 3:54 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits