Change #1171270 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] cirrussearch: move cirrussearch20(89|91) to insetup

https://gerrit.wikimedia.org/r/1171270

Change #1171270 merged by Bking:

[operations/puppet@production] cirrussearch: move cirrussearch20(89|91) to insetup

https://gerrit.wikimedia.org/r/1171270

@bking looks like cirrussearch2089 won't boot at all. it is under warranty, so I'm starting the process with Dell. It's probably the backplane or the MB that failed so it might take some time to get that one back up. They don't like replacing those.

As for cirrussearch2091, by every measure i can check, PXE should be working. rant he provisioning script to catch any errant settings (with the uefi tag since bios was already set that way) NIC is working and showing connection to the correct switch port.
The error that's coming up on console is: PXE-E18 Server Response Timeout. Ruled out secure boot issue. updated the firmware. replaced the cable.

Could you check the DNS settings? it's the only other thing I'm finding that might be the issue. My search returned vague results, so I'm not sure what you should be looking for. ty!

@Jhancock.wm following up on our IRC discussion yesterday, I've already spent hours troubleshooting cirrussearch2091 and unfortunately, I'm out of my depth. If you and I aren't able to fix it, then it sounds like we need to engage Dell? If I can help with that, or if there is a different escalation path, please let me know.

@bking john took a look at cirrusearch2091 for me and it looks like he was able to get it to PXE using legacy BIOS instead of UEFI. Will that work for this server?

@Jhancock.wm sure, can y'all try installing Bullseye on it? I only switched it to UEFI because it was failing to PXE boot with legacy BIOS. I tried at least 5 or 6 times with legacy BIOS, it would get to the PXE screen and say it was booting, but never actually get to the Debian installer.

If it works, great! It wouldn't be the first time @Jclark-ctr worked magic on one of our hosts ;) .

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin1003 for host cirrussearch2091.codfw.wmnet with OS bullseye

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin1003 for host cirrussearch2091.codfw.wmnet with OS bullseye completed:

• cirrussearch2091 (PASS)
  • Removed from Puppet and PuppetDB if present and deleted any certificates
  • Removed from Debmonitor if present
  • Forced PXE for next reboot
  • Host rebooted via IPMI
  • Host up (Debian installer)
  • Add puppet_version metadata (7) to Debian installer
  • Checked BIOS boot parameters are back to normal
  • Host up (new fresh bullseye OS)
  • Generated Puppet certificate
  • Signed new Puppet certificate
  • Run Puppet in NOOP mode to populate exported resources in PuppetDB
  • Found Nagios_host resource for this host in PuppetDB
  • Downtimed the new host on Icinga/Alertmanager
  • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202507241444_jhancock_3228538_cirrussearch2091.out
  • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
  • Rebooted
  • Automatic Puppet run was successful
  • Forced a re-check of all Icinga services for the host
  • Icinga status is optimal
  • Icinga downtime removed
  • Updated Netbox data from PuppetDB

@jhathaway that one took. Still working with Dell on the other server. I'll let you know when progress is made.

@Jhancock.wm I think you meant to ping me instead of Jesse ;). No worries either way though, I've created T400640 to put cirrussearch2091 back into service. Thanks again for your help!