Page MenuHomePhabricator
Create Task
Maniphest T408752

Q2:rack/setup/install wikikube-worker1335-59
Closed, ResolvedPublic
Actions

Description

This task will track the racking, setup, and OS installation of

Hostname / Racking / Installation Details

This section should list the racking restrictions for these hosts. If they shouldn't share a rack/row with one another, or any existing hosts. This section should also list the other details listed below.

Hostnames: What are the hostnames, and have you updated https://wikitech.wikimedia.org/wiki/Infrastructure_naming_conventions ? wikikube-worker13[35-59]
Racking Proposal: Where should these systems be racked? Can they share with any existing systems or should they avoid any other systems sharing their rack or row? Use https://fault-tolerance.toolforge.org/map to optimize this placement.

  • Row A: 0
  • Row B: 0
  • Row C: 8
  • Row D: 7
  • Row E: 4
  • Row F: 6

Networking Setup: # of Connections:1/2 - Speed:1G/10G. - VLAN:Private/Public/Other(Specify) :
OS Distro: Bookworm (default unless otherwise specified)
Boot Method: Legacy BIOS or UEFI. Please note UEFI must have partman updates applied in advance of setup and is currently in pilot program: https://wikitech.wikimedia.org/wiki/UEFI_Boot
Sub-team Technical Contact: Who should our on-sites contact with any questions involving system racking and setup? @Clement_Goubert

Per host setup checklist

Each host should have its own setup checklist copied and pasted into the list below.

wikikube-worker1335
  • Receive in system on procurement task T405284 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
wikikube-worker1336
  • Receive in system on procurement task T405284 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
wikikube-worker1337
  • Receive in system on procurement task T405284 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
wikikube-worker1338
  • Receive in system on procurement task T405284 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
wikikube-worker1339
  • Receive in system on procurement task T405284 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
wikikube-worker1340
  • Receive in system on procurement task T405284 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
wikikube-worker1341
  • Receive in system on procurement task T405284 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
wikikube-worker1342
  • Receive in system on procurement task T405284 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
wikikube-worker1343
  • Receive in system on procurement task T405284 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
wikikube-worker1344
  • Receive in system on procurement task T405284 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
wikikube-worker1345
  • Receive in system on procurement task T405284 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
wikikube-worker1346
  • Receive in system on procurement task T405284 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
wikikube-worker1347
  • Receive in system on procurement task T405284 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
wikikube-worker1348
  • Receive in system on procurement task T405284 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
wikikube-worker1349
  • Receive in system on procurement task T405284 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
wikikube-worker1350
  • Receive in system on procurement task T405284 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
wikikube-worker1351
  • Receive in system on procurement task T405284 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
wikikube-worker1352
  • Receive in system on procurement task T405284 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
wikikube-worker1353
  • Receive in system on procurement task T405284 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
wikikube-worker1354
  • Receive in system on procurement task T405284 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
wikikube-worker1355
  • Receive in system on procurement task T405284 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
wikikube-worker1356
  • Receive in system on procurement task T405284 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
wikikube-worker1357
  • Receive in system on procurement task T405284 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
wikikube-worker1358
  • Receive in system on procurement task T405284 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
wikikube-worker1359
  • Receive in system on procurement task T405284 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
partman: New wikikube-worker need UEFIoperations/puppetproduction+9 -0
site.pp: Add new wikikube insetup hostsoperations/puppetproduction+20 -0
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
Unknown Object (Task)
 ResolvedJclark-ctrT408752 Q2:rack/setup/install wikikube-worker1335-59

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Maintenance_bot removed a project: Patch-For-Review.Dec 22 2025, 4:30 PM
Jclark-ctr claimed this task.Jan 6 2026, 10:01 PM
Jclark-ctr updated the task description. (Show Details)
Jclark-ctr added a comment.Jan 8 2026, 7:36 AM

@Clement_Goubert Before I start racking these, do you want to verify that they’re correct by row, since we had so many orders for these?

Clement_Goubert added a comment.Jan 8 2026, 4:38 PM
In T408752#11502255, @Jclark-ctr wrote:

@Clement_Goubert Before I start racking these, do you want to verify that they’re correct by row, since we had so many orders for these?

This one's pretty set, we're catching up rows C-F to rows A-B. I'm noting looking at https://fault-tolerance.toolforge.org/map?cluster=kubernetes that wikikube hosts are pretty concentrated to specific racks in rows C and D (none in C1, C2, C4, C7, C8, D2, D4, D5, D6, D7) so if you have the opportunity to rack some in these, that would help with rack-level resilience.

Jclark-ctr updated the task description. (Show Details)Jan 12 2026, 2:41 PM

All servers racked and netbox configured holding for update on 4ft power cables

ops-monitoring-bot added a comment.Jan 13 2026, 12:49 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1003 for host wikikube-worker1335.eqiad.wmnet with OS trixie

ops-monitoring-bot added a comment.Jan 13 2026, 12:51 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1003 for host wikikube-worker1336.eqiad.wmnet with OS trixie

ops-monitoring-bot added a comment.Jan 13 2026, 12:53 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1003 for host wikikube-worker1337.eqiad.wmnet with OS trixie

ops-monitoring-bot added a comment.Jan 13 2026, 12:56 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1003 for host wikikube-worker1340.eqiad.wmnet with OS trixie

ops-monitoring-bot added a comment.Jan 13 2026, 12:56 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1003 for host wikikube-worker1339.eqiad.wmnet with OS trixie

ops-monitoring-bot added a comment.Jan 13 2026, 12:57 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1003 for host wikikube-worker1341.eqiad.wmnet with OS trixie

Jclark-ctr updated the task description. (Show Details)Jan 13 2026, 12:59 PM
ops-monitoring-bot added a comment.Jan 13 2026, 1:25 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1003 for host wikikube-worker1335.eqiad.wmnet with OS trixie completed:

  • wikikube-worker1335 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202601131304_jclark_1244016_wikikube-worker1335.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Jan 13 2026, 1:26 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1003 for host wikikube-worker1336.eqiad.wmnet with OS trixie completed:

  • wikikube-worker1336 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202601131307_jclark_1246179_wikikube-worker1336.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Jan 13 2026, 1:28 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1003 for host wikikube-worker1339.eqiad.wmnet with OS trixie completed:

  • wikikube-worker1339 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202601131311_jclark_1246478_wikikube-worker1339.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Jan 13 2026, 1:32 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1003 for host wikikube-worker1342.eqiad.wmnet with OS trixie

ops-monitoring-bot added a comment.Jan 13 2026, 1:32 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1003 for host wikikube-worker1343.eqiad.wmnet with OS trixie

ops-monitoring-bot added a comment.Jan 13 2026, 1:32 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1003 for host wikikube-worker1344.eqiad.wmnet with OS trixie

Jclark-ctr updated the task description. (Show Details)Jan 13 2026, 1:33 PM
ops-monitoring-bot added a comment.Jan 13 2026, 1:34 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1003 for host wikikube-worker1340.eqiad.wmnet with OS trixie completed:

  • wikikube-worker1340 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202601131317_jclark_1246472_wikikube-worker1340.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Jan 13 2026, 1:35 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1003 for host wikikube-worker1345.eqiad.wmnet with OS trixie

ops-monitoring-bot added a comment.Jan 13 2026, 1:37 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1003 for host wikikube-worker1341.eqiad.wmnet with OS trixie completed:

  • wikikube-worker1341 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202601131321_jclark_1246567_wikikube-worker1341.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Jan 13 2026, 1:38 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1003 for host wikikube-worker1346.eqiad.wmnet with OS trixie

Jclark-ctr updated the task description. (Show Details)Jan 13 2026, 1:39 PM
ops-monitoring-bot added a comment.Jan 13 2026, 1:46 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1003 for host wikikube-worker1337.eqiad.wmnet with OS trixie completed:

  • wikikube-worker1337 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202601131329_jclark_1246262_wikikube-worker1337.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Jan 13 2026, 2:05 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1003 for host wikikube-worker1342.eqiad.wmnet with OS trixie completed:

  • wikikube-worker1342 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202601131349_jclark_1258395_wikikube-worker1342.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
Jclark-ctr updated the task description. (Show Details)Jan 13 2026, 2:06 PM
ops-monitoring-bot added a comment.Jan 13 2026, 2:08 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1003 for host wikikube-worker1343.eqiad.wmnet with OS trixie completed:

  • wikikube-worker1343 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202601131352_jclark_1258404_wikikube-worker1343.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
Jclark-ctr updated the task description. (Show Details)Jan 13 2026, 2:09 PM
ops-monitoring-bot added a comment.Jan 13 2026, 2:13 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1003 for host wikikube-worker1344.eqiad.wmnet with OS trixie completed:

  • wikikube-worker1344 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202601131356_jclark_1258409_wikikube-worker1344.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
Jclark-ctr updated the task description. (Show Details)Jan 13 2026, 2:13 PM
ops-monitoring-bot added a comment.Jan 13 2026, 2:16 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1003 for host wikikube-worker1338.eqiad.wmnet with OS trixie

ops-monitoring-bot added a comment.Jan 13 2026, 2:17 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1003 for host wikikube-worker1345.eqiad.wmnet with OS trixie completed:

  • wikikube-worker1345 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202601131400_jclark_1258746_wikikube-worker1345.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Jan 13 2026, 2:18 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1003 for host wikikube-worker1349.eqiad.wmnet with OS trixie

ops-monitoring-bot added a comment.Jan 13 2026, 2:18 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1003 for host wikikube-worker1350.eqiad.wmnet with OS trixie

ops-monitoring-bot added a comment.Jan 13 2026, 2:18 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1003 for host wikikube-worker1351.eqiad.wmnet with OS trixie

ops-monitoring-bot added a comment.Jan 13 2026, 2:21 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1003 for host wikikube-worker1346.eqiad.wmnet with OS trixie completed:

  • wikikube-worker1346 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202601131403_jclark_1259366_wikikube-worker1346.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Jan 13 2026, 2:22 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1003 for host wikikube-worker1352.eqiad.wmnet with OS trixie

ops-monitoring-bot added a comment.Jan 13 2026, 2:29 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1003 for host wikikube-worker1348.eqiad.wmnet with OS trixie

ops-monitoring-bot added a comment.Jan 13 2026, 2:29 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1003 for host wikikube-worker1347.eqiad.wmnet with OS trixie

Jclark-ctr updated the task description. (Show Details)Jan 13 2026, 2:48 PM
Jclark-ctr updated the task description. (Show Details)
ops-monitoring-bot added a comment.Jan 13 2026, 2:52 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1003 for host wikikube-worker1338.eqiad.wmnet with OS trixie completed:

  • wikikube-worker1338 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202601131434_jclark_1275531_wikikube-worker1338.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
Jclark-ctr updated the task description. (Show Details)Jan 13 2026, 2:53 PM
ops-monitoring-bot added a comment.Jan 13 2026, 2:55 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1003 for host wikikube-worker1350.eqiad.wmnet with OS trixie completed:

  • wikikube-worker1350 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202601131438_jclark_1276070_wikikube-worker1350.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
Jclark-ctr updated the task description. (Show Details)Jan 13 2026, 2:55 PM
JMeybohm subscribed.Jan 13 2026, 2:57 PM
ops-monitoring-bot added a comment.Jan 13 2026, 2:59 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1003 for host wikikube-worker1349.eqiad.wmnet with OS trixie completed:

  • wikikube-worker1349 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202601131441_jclark_1276036_wikikube-worker1349.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
Jclark-ctr updated the task description. (Show Details)Jan 13 2026, 3:00 PM
ops-monitoring-bot added a comment.Jan 13 2026, 3:06 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1003 for host wikikube-worker1351.eqiad.wmnet with OS trixie completed:

  • wikikube-worker1351 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202601131449_jclark_1276118_wikikube-worker1351.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Jan 13 2026, 3:10 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1003 for host wikikube-worker1348.eqiad.wmnet with OS trixie completed:

  • wikikube-worker1348 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202601131452_jclark_1280679_wikikube-worker1348.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Jan 13 2026, 3:11 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1003 for host wikikube-worker1359.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Jan 13 2026, 3:11 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1003 for host wikikube-worker1358.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Jan 13 2026, 3:11 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1003 for host wikikube-worker1357.eqiad.wmnet with OS bookworm

Jclark-ctr updated the task description. (Show Details)Jan 13 2026, 3:12 PM
ops-monitoring-bot added a comment.Jan 13 2026, 3:13 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1003 for host wikikube-worker1352.eqiad.wmnet with OS trixie completed:

  • wikikube-worker1352 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202601131455_jclark_1278610_wikikube-worker1352.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
Jclark-ctr updated the task description. (Show Details)Jan 13 2026, 3:15 PM
ops-monitoring-bot added a comment.Jan 13 2026, 3:17 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1003 for host wikikube-worker1354.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Jan 13 2026, 3:18 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1003 for host wikikube-worker1355.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Jan 13 2026, 3:18 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1003 for host wikikube-worker1356.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Jan 13 2026, 3:29 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1003 for host wikikube-worker1347.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Jan 13 2026, 3:46 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1003 for host wikikube-worker1359.eqiad.wmnet with OS bookworm completed:

  • wikikube-worker1359 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202601131529_jclark_1298253_wikikube-worker1359.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
Jclark-ctr updated the task description. (Show Details)Jan 13 2026, 3:47 PM
ops-monitoring-bot added a comment.Jan 13 2026, 3:51 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1003 for host wikikube-worker1354.eqiad.wmnet with OS bookworm completed:

  • wikikube-worker1354 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202601131533_jclark_1300675_wikikube-worker1354.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
Jclark-ctr updated the task description. (Show Details)Jan 13 2026, 3:51 PM
ops-monitoring-bot added a comment.Jan 13 2026, 3:55 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1003 for host wikikube-worker1357.eqiad.wmnet with OS bookworm completed:

  • wikikube-worker1357 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202601131537_jclark_1298434_wikikube-worker1357.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
Jclark-ctr updated the task description. (Show Details)Jan 13 2026, 3:55 PM
ops-monitoring-bot added a comment.Jan 13 2026, 3:57 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1003 for host wikikube-worker1356.eqiad.wmnet with OS bookworm completed:

  • wikikube-worker1356 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202601131541_jclark_1301019_wikikube-worker1356.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
Jclark-ctr updated the task description. (Show Details)Jan 13 2026, 3:58 PM
ops-monitoring-bot added a comment.Jan 13 2026, 4:05 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1003 for host wikikube-worker1347.eqiad.wmnet with OS bookworm completed:

  • wikikube-worker1347 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Unable to disable Puppet, the host may have been unreachable
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202601131548_jclark_1304671_wikikube-worker1347.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Jan 13 2026, 4:05 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1003 for host wikikube-worker1347.eqiad.wmnet with OS bookworm executed with errors:

  • wikikube-worker1347 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Unable to disable Puppet, the host may have been unreachable
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202601131548_jclark_1304671_wikikube-worker1347.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console wikikube-worker1347.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
Jclark-ctr updated the task description. (Show Details)Jan 13 2026, 4:06 PM
Jclark-ctr added a comment.Jan 13 2026, 4:10 PM

wikikube-worker1353 is complaining of cable disconnected
wikikube-worker1355 is failing to image will need to investigate

ops-monitoring-bot added a comment.Jan 13 2026, 4:11 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1003 for host wikikube-worker1358.eqiad.wmnet with OS bookworm completed:

  • wikikube-worker1358 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202601131553_jclark_1298347_wikikube-worker1358.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
Jclark-ctr updated the task description. (Show Details)Jan 13 2026, 4:11 PM
ops-monitoring-bot added a comment.Jan 13 2026, 4:38 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1003 for host wikikube-worker1355.eqiad.wmnet with OS bookworm executed with errors:

  • wikikube-worker1355 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console wikikube-worker1355.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
Jclark-ctr mentioned this in T414496: Power Supply - PS1 Status - issue on wikikube-worker1338:9290.Jan 13 2026, 6:46 PM
ops-monitoring-bot added a comment.Jan 13 2026, 8:47 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1003 for host wikikube-worker1353.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Jan 13 2026, 8:47 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1003 for host wikikube-worker1355.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Jan 13 2026, 8:48 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1003 for host wikikube-worker1352.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Jan 13 2026, 9:01 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1003 for host wikikube-worker1353.eqiad.wmnet with OS bookworm executed with errors:

  • wikikube-worker1353 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console wikikube-worker1353.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Jan 13 2026, 9:02 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1003 for host wikikube-worker1352.eqiad.wmnet with OS bookworm executed with errors:

  • wikikube-worker1352 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console wikikube-worker1352.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Jan 13 2026, 9:11 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1003 for host wikikube-worker1353.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Jan 13 2026, 9:11 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1003 for host wikikube-worker1352.eqiad.wmnet with OS bookworm

Jclark-ctr updated the task description. (Show Details)Jan 13 2026, 9:20 PM
ops-monitoring-bot added a comment.Jan 13 2026, 9:20 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1003 for host wikikube-worker1352.eqiad.wmnet with OS bookworm executed with errors:

  • wikikube-worker1352 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console wikikube-worker1352.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Jan 13 2026, 9:21 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1003 for host wikikube-worker1353.eqiad.wmnet with OS bookworm executed with errors:

  • wikikube-worker1353 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console wikikube-worker1353.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Jan 13 2026, 9:21 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1003 for host wikikube-worker1355.eqiad.wmnet with OS bookworm completed:

  • wikikube-worker1355 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202601132104_jclark_1348824_wikikube-worker1355.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Jan 13 2026, 9:23 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1003 for host wikikube-worker1353.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Jan 13 2026, 9:25 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1003 for host wikikube-worker1352.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Jan 13 2026, 9:51 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1003 for host wikikube-worker1352.eqiad.wmnet with OS bookworm completed:

  • wikikube-worker1352 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202601132134_jclark_1357354_wikikube-worker1352.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Jan 13 2026, 9:54 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1003 for host wikikube-worker1353.eqiad.wmnet with OS bookworm executed with errors:

  • wikikube-worker1353 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Generated Puppet certificate
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console wikikube-worker1353.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Jan 13 2026, 11:30 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1003 for host wikikube-worker1353.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Jan 13 2026, 11:55 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1003 for host wikikube-worker1353.eqiad.wmnet with OS bookworm completed:

  • wikikube-worker1353 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202601132338_jclark_1381525_wikikube-worker1353.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
Jclark-ctr closed this task as Resolved.Jan 13 2026, 11:56 PM
Jclark-ctr updated the task description. (Show Details)
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits