Page MenuHomePhabricator
Create Task
Maniphest T399964

Q1:rack/setup/install an-worker12[09-32].eqiad.wmnet
Closed, ResolvedPublic
Actions

Description

This task will track the racking, setup, and OS installation of an-worker12[09-32].eqiad.wmnet

Hostname / Racking / Installation Details

Hostnames: an-worker12[09-32].eqiad.wmnet
Racking Proposal: Spread across eqiad rows A-F, trying not to be imbalanced in terms of any one rack or row. Current distribution is here.
Networking Setup: # of Connections:1- Speed:10G. - VLAN:analytics
OS Distro: Bullseye
Boot Method: Legacy BIOS
Sub-team Technical Contact: @BTullis

Per host setup checklist

Each host should have its own setup checklist copied and pasted into the list below.

an-worker1209
  • Receive in system on procurement task T398467 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
an-worker1210
  • Receive in system on procurement task T398467 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
an-worker1211
  • Receive in system on procurement task T398467 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
an-worker1212
  • Receive in system on procurement task T398467 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
an-worker1213
  • Receive in system on procurement task T398467 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
an-worker1214
  • Receive in system on procurement task T398467 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
an-worker1215
  • Receive in system on procurement task T398467 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
an-worker1216
  • Receive in system on procurement task T398467 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
an-worker1217
  • Receive in system on procurement task T398467 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
an-worker1218
  • Receive in system on procurement task T398467 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
an-worker1219
  • Receive in system on procurement task T398467 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
an-worker1220
  • Receive in system on procurement task T398467 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
an-worker1221
  • Receive in system on procurement task T398467 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
an-worker1222
  • Receive in system on procurement task T398467 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
an-worker1223
  • Receive in system on procurement task T398467 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
an-worker1224
  • Receive in system on procurement task T398467 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
an-worker1225
  • Receive in system on procurement task T398467 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
an-worker1226
  • Receive in system on procurement task T398467 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
an-worker1227
  • Receive in system on procurement task T398467 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
an-worker1228
  • Receive in system on procurement task T398467 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
an-worker1229
  • Receive in system on procurement task T398467 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
an-worker1230
  • Receive in system on procurement task T398467 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
an-worker1231
  • Receive in system on procurement task T398467 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
an-worker1232
  • Receive in system on procurement task T398467 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
Correct the preseed value for the new an-worker nodesoperations/puppetproduction+1 -1
Add the new an-worker nodes to site.pp and preseed.yamloperations/puppetproduction+6 -0
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
Unknown Object (Task)
 ResolvedJclark-ctrT399964 Q1:rack/setup/install an-worker12[09-32].eqiad.wmnet
 ResolvedBTullisT405903 Bring an-worker12[09-32] into service

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Jclark-ctr subscribed.Aug 19 2025, 7:05 PM

@BTullis do we have any idea of when the EOL an-workers will be Decommissioned? We are very tight on space and power in 10g racks and we already have majority in rows E/F

BTullis moved this task from Done to Reported on the Data-Platform-SRE (2025.07.26 - 2025.08.15) board.Aug 20 2025, 10:32 AM
BTullis added subscribers: Ahoelzl, brouberol, xcollazo and 3 others.Aug 20 2025, 4:07 PM

Hi @Jclark-ctr - It's a bit complicated - sorry :-)

We are currently in the middle of re-assessing whether or not we are going to go ahead with a project to upgrade Hadoop.

See these links for more details: T379385: Upgrade Hadoop to version 3.3.6 and Hive to version 4.0.1

If we decide to go ahead with the upgrade, then the 46 workers identified here will be needed for a few more months: T397166: Reimage and rename 46 hadoop worker nodes to use in the HDFS backup cluster
This is because we plan to use them to create a temporary HDFS backup cluster to support the in-place upgrade of the main HDFS cluster. After the successful upgrade the backup cluster and all of its hosts will be decommissioned.

However, if we decide that we are not going to go ahead with an upgrade to Hadoop 3, then all 46 hosts (and the two in T397175) can be decommissioned straight away.

We aren't even sure, at this rate, whether these 24 new hosts will end up being an-worker12[09-32] (as originally specified) or whether they will be re-named and used as dse-k8s-worker nodes, or something else.
The racking plan for these new hosts might even change as a result of this strategic decision-making, depending on whether they stay as Hadoop workers, or become Kubernetes workers.

So I'm sorry to be difficult. We are hoping to be able to make that decision within the first week of September and will let you know asap.
I'm tagging some folk (@Ahoelzl, @JAllemandou, @brouberol, @xcollazo, @tchin, @mforns) who are all involved in the decision making.

If you need to press for further information during the next week, then @Ahoelzl would be the best person to contact.

Jclark-ctr updated the task description. (Show Details)Aug 21 2025, 2:16 PM
Jclark-ctr claimed this task.Aug 22 2025, 12:00 PM
Jclark-ctr moved this task from Racking Tasks to Blocked on the ops-eqiad board.
Jclark-ctr added a comment.Aug 22 2025, 1:09 PM

All Devices added to Netbox

Jclark-ctr moved this task from Blocked to Racking Tasks on the ops-eqiad board.Sep 23 2025, 1:20 PM
Jclark-ctr updated the task description. (Show Details)Sep 24 2025, 1:54 PM
Jclark-ctr updated the task description. (Show Details)Sep 24 2025, 11:55 PM
ops-monitoring-bot added a comment.Sep 25 2025, 1:15 AM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host an-worker1210.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 25 2025, 1:15 AM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host an-worker1210.eqiad.wmnet with OS bullseye executed with errors:

  • an-worker1210 (FAIL)
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console an-worker1210.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
Jclark-ctr updated the task description. (Show Details)Sep 25 2025, 1:17 AM
ops-monitoring-bot added a comment.Sep 25 2025, 1:20 AM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host an-worker1210.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 25 2025, 1:20 AM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host an-worker1210.eqiad.wmnet with OS bullseye executed with errors:

  • an-worker1210 (FAIL)
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console an-worker1210.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
Jclark-ctr added a comment.Sep 25 2025, 1:36 AM

@BTullis I’ve only set up RAID1 for an-worker1210. I wanted to get one running before the end of the night, but I’m not having any luck. Could you help me with this error unable to find hostcert?

  • OUTPUT of 'puppet node clea...1210.eqiad.wmnet' -----

Error: Could not find 'hostcert' at '/var/lib/puppet/ssl/certs/puppetserver1001.eqiad.wmnet.pem'

Error: Try 'puppet help node clean' for usage

100.0% (1/1) of nodes failed to execute command 'puppet node clea...1210.eqiad.wmnet': puppetserver1001.eqiad.wmnet
0.0% (0/1) success ratio (< 100.0% threshold) for command: 'puppet node clea...1210.eqiad.wmnet'. Aborting.
0.0% (0/1) success ratio (< 100.0% threshold) of nodes successfully executed all commands. Aborting.

ops-monitoring-bot added a comment.Sep 26 2025, 1:51 AM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host an-worker1231.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 26 2025, 2:16 AM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host an-worker1230.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 26 2025, 2:16 AM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host an-worker1232.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 26 2025, 2:16 AM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host an-worker1229.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 26 2025, 2:19 AM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host an-worker1228.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 26 2025, 2:19 AM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host an-worker1227.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 26 2025, 2:19 AM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host an-worker1226.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 26 2025, 2:20 AM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host an-worker1225.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 26 2025, 2:36 AM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host an-worker1227.eqiad.wmnet with OS bullseye executed with errors:

  • an-worker1227 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console an-worker1227.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Sep 26 2025, 2:36 AM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host an-worker1225.eqiad.wmnet with OS bullseye executed with errors:

  • an-worker1225 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console an-worker1225.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Sep 26 2025, 2:36 AM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host an-worker1226.eqiad.wmnet with OS bullseye executed with errors:

  • an-worker1226 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console an-worker1226.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Sep 26 2025, 2:37 AM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host an-worker1229.eqiad.wmnet with OS bullseye executed with errors:

  • an-worker1229 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console an-worker1229.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Sep 26 2025, 2:38 AM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host an-worker1231.eqiad.wmnet with OS bullseye completed:

  • an-worker1231 (WARN)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202509260214_jclark_3237393_an-worker1231.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Sep 26 2025, 2:58 AM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host an-worker1232.eqiad.wmnet with OS bullseye completed:

  • an-worker1232 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202509260238_jclark_3260789_an-worker1232.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Sep 26 2025, 2:59 AM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host an-worker1230.eqiad.wmnet with OS bullseye completed:

  • an-worker1230 (WARN)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202509260234_jclark_3260838_an-worker1230.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
Jclark-ctr updated the task description. (Show Details)Sep 26 2025, 2:59 AM

found a number of servers not imaging believe this is that cause

'an-worker1209|an-worker121[0-9]an-worker122[0-9]|an-worker123[0-2]'

missing a pipe between an-worker121[0-9] and an-worker122[0-9].

It should be

'an-worker1209|an-worker121[0-9]|an-worker122[0-9]|an-worker123[0-2]'

gerritbot added a comment.Sep 26 2025, 10:19 AM

Change #1191667 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] Correct the preseed value for the new an-worker nodes

https://gerrit.wikimedia.org/r/1191667

gerritbot added a project: Patch-For-Review.Sep 26 2025, 10:19 AM
gerritbot added a comment.Sep 26 2025, 10:24 AM

Change #1191667 merged by Btullis:

[operations/puppet@production] Correct the preseed value for the new an-worker nodes

https://gerrit.wikimedia.org/r/1191667

Maintenance_bot removed a project: Patch-For-Review.Sep 26 2025, 10:32 AM
ops-monitoring-bot added a comment.Sep 26 2025, 12:36 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host an-worker1229.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 26 2025, 12:47 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host an-worker1228.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 26 2025, 12:49 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host an-worker1227.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 26 2025, 12:50 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host an-worker1226.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 26 2025, 12:51 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host an-worker1225.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 26 2025, 1:09 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host an-worker1225.eqiad.wmnet with OS bullseye executed with errors:

  • an-worker1225 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console an-worker1225.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Sep 26 2025, 1:14 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host an-worker1225.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 26 2025, 1:21 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host an-worker1229.eqiad.wmnet with OS bullseye completed:

  • an-worker1229 (WARN)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202509261257_jclark_3845102_an-worker1229.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Sep 26 2025, 1:28 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host an-worker1228.eqiad.wmnet with OS bullseye completed:

  • an-worker1228 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202509261309_jclark_3854957_an-worker1228.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Sep 26 2025, 1:42 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host an-worker1226.eqiad.wmnet with OS bullseye completed:

  • an-worker1226 (WARN)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202509261318_jclark_3857391_an-worker1226.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Sep 26 2025, 1:58 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host an-worker1227.eqiad.wmnet with OS bullseye completed:

  • an-worker1227 (WARN)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202509261313_jclark_3855911_an-worker1227.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • Failed to run the sre.puppet.sync-netbox-hiera cookbook, run it manually
ops-monitoring-bot added a comment.Sep 26 2025, 1:59 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host an-worker1225.eqiad.wmnet with OS bullseye completed:

  • an-worker1225 (WARN)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202509261333_jclark_3882283_an-worker1225.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Sep 26 2025, 5:47 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host an-worker1224.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 26 2025, 5:47 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host an-worker1223.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 26 2025, 5:48 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host an-worker1222.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 26 2025, 5:48 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host an-worker1221.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 26 2025, 5:48 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host an-worker1220.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 26 2025, 6:33 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host an-worker1224.eqiad.wmnet with OS bullseye completed:

  • an-worker1224 (WARN)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202509261809_jclark_29547_an-worker1224.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Sep 26 2025, 6:35 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host an-worker1222.eqiad.wmnet with OS bullseye executed with errors:

  • an-worker1222 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202509261813_jclark_29747_an-worker1222.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console an-worker1222.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Sep 26 2025, 6:41 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host an-worker1221.eqiad.wmnet with OS bullseye completed:

  • an-worker1221 (WARN)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202509261816_jclark_29801_an-worker1221.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Sep 26 2025, 6:45 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host an-worker1223.eqiad.wmnet with OS bullseye completed:

  • an-worker1223 (WARN)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202509261820_jclark_29684_an-worker1223.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Sep 26 2025, 6:51 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host an-worker1220.eqiad.wmnet with OS bullseye completed:

  • an-worker1220 (WARN)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202509261823_jclark_29873_an-worker1220.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Sep 26 2025, 6:55 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host an-worker1219.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 26 2025, 6:56 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host an-worker1218.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 26 2025, 6:58 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host an-worker1217.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 26 2025, 7:02 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host an-worker1215.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 26 2025, 7:02 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host an-worker1216.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 26 2025, 7:07 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host an-worker1214.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 26 2025, 7:07 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host an-worker1213.eqiad.wmnet with OS bullseye

xcollazo unsubscribed.Sep 26 2025, 7:16 PM
ops-monitoring-bot added a comment.Sep 26 2025, 7:22 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host an-worker1214.eqiad.wmnet with OS bullseye executed with errors:

  • an-worker1214 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console an-worker1214.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Sep 26 2025, 7:22 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host an-worker1213.eqiad.wmnet with OS bullseye executed with errors:

  • an-worker1213 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console an-worker1213.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Sep 26 2025, 7:43 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host an-worker1217.eqiad.wmnet with OS bullseye completed:

  • an-worker1217 (WARN)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202509261917_jclark_104032_an-worker1217.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Sep 26 2025, 7:46 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host an-worker1215.eqiad.wmnet with OS bullseye completed:

  • an-worker1215 (WARN)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202509261921_jclark_111443_an-worker1215.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Sep 26 2025, 7:47 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host an-worker1216.eqiad.wmnet with OS bullseye completed:

  • an-worker1216 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202509261929_jclark_105104_an-worker1216.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Sep 26 2025, 7:50 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host an-worker1218.eqiad.wmnet with OS bullseye completed:

  • an-worker1218 (WARN)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202509261925_jclark_103722_an-worker1218.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Sep 26 2025, 7:51 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host an-worker1219.eqiad.wmnet with OS bullseye completed:

  • an-worker1219 (WARN)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202509261914_jclark_103268_an-worker1219.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • Failed to run the sre.puppet.sync-netbox-hiera cookbook, run it manually
ops-monitoring-bot added a comment.Sep 26 2025, 7:55 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host an-worker1214.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 26 2025, 7:55 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host an-worker1213.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 26 2025, 7:56 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host an-worker1212.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 26 2025, 7:57 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host an-worker1211.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 26 2025, 7:57 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host an-worker1209.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 26 2025, 7:57 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host an-worker1210.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 26 2025, 8:33 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host an-worker1213.eqiad.wmnet with OS bullseye completed:

  • an-worker1213 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202509262013_jclark_168306_an-worker1213.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Sep 26 2025, 8:35 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host an-worker1209.eqiad.wmnet with OS bullseye completed:

  • an-worker1209 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202509262016_jclark_170460_an-worker1209.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Sep 26 2025, 8:41 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host an-worker1211.eqiad.wmnet with OS bullseye completed:

  • an-worker1211 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202509262023_jclark_169466_an-worker1211.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Sep 26 2025, 8:45 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host an-worker1212.eqiad.wmnet with OS bullseye completed:

  • an-worker1212 (WARN)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202509262020_jclark_169028_an-worker1212.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Sep 26 2025, 8:52 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host an-worker1210.eqiad.wmnet with OS bullseye completed:

  • an-worker1210 (WARN)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202509262027_jclark_170278_an-worker1210.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Sep 26 2025, 9:08 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host an-worker1214.eqiad.wmnet with OS bullseye executed with errors:

  • an-worker1214 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console an-worker1214.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Sep 26 2025, 9:13 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host an-worker1214.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 26 2025, 9:56 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host an-worker1214.eqiad.wmnet with OS bullseye completed:

  • an-worker1214 (WARN)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202509262132_jclark_251901_an-worker1214.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
Jclark-ctr updated the task description. (Show Details)Sep 27 2025, 11:12 PM
Jclark-ctr reassigned this task from Jclark-ctr to BTullis.Sep 27 2025, 11:16 PM

Hey Ben everything has been imaged If you can assist with running perccli64 command

perccli64 /c0 add vd each r0 wb ra

because access has not been resolved yet for T399964

BTullis added a comment.Sep 29 2025, 11:36 AM

That worked well, it seems.

btullis@cumin1003:~$ sudo cumin 'an-worker[1209-1232].eqiad.wmnet' 'perccli64 /c0 add vd each r0 wb ra'
24 hosts will be targeted:
an-worker[1209-1232].eqiad.wmnet
OK to proceed on 24 hosts? Enter the number of affected hosts to confirm or "q" to quit: 24
===== NODE GROUP =====                                                                                                                                                                                             
(24) an-worker[1209-1232].eqiad.wmnet                                                                                                                                                                              
----- OUTPUT of 'perccli64 /c0 add vd each r0 wb ra' -----                                                                                                                                                         
CLI Version = 007.1910.0000.0000 Oct 08, 2021                                                                                                                                                                      
Operating system = Linux 5.10.0-35-amd64                                                                                                                                                                           
Controller = 0
Status = Success
Description = Add VD Succeeded.


================                                                                                                                                                                                                   
PASS |███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 100% (24/24) [00:47<00:00,  1.97s/hosts]
FAIL |                                                                                                                                                                            |   0% (0/24) [00:47<?, ?hosts/s]
100.0% (24/24) success ratio (>= 100.0% threshold) for command: 'perccli64 /c0 add vd each r0 wb ra'.
100.0% (24/24) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
btullis@cumin1003:~$
BTullis closed this task as Resolved.Sep 29 2025, 11:42 AM

I think we can resolve this now. I have created T405903 to track adding these hosts to the cluster.

Jclark-ctr claimed this task.Sep 29 2025, 2:42 PM
BTullis closed subtask T405903: Bring an-worker12[09-32] into service as Resolved.Sep 30 2025, 10:01 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits