Change #1280373 had a related patch set uploaded (by MVernon; author: MVernon):

[operations/puppet@production] swift: prep for ms-be11*

https://gerrit.wikimedia.org/r/1280373

Change #1280373 merged by MVernon:

[operations/puppet@production] swift: prep for ms-be11*

https://gerrit.wikimedia.org/r/1280373

Cookbook cookbooks.sre.hosts.reimage was started by vriley@cumin1003 for host ms-be1098.eqiad.wmnet with OS bullseye

@MatthewVernon I wanted to check on something with these servers. It seems as though the cards are 25 gig and it's not able to run through the re-image. Also, is there a specific RAID we should be using for these units?

Cookbook cookbooks.sre.hosts.reimage started by vriley@cumin1003 for host ms-be1098.eqiad.wmnet with OS bullseye executed with errors:

• ms-be1098 (FAIL)
  • Removed from Puppet and PuppetDB if present and deleted any certificates
  • Removed from Debmonitor if present
  • Forced UEFI HTTP Boot for next reboot
  • Host rebooted via Redfish
  • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console ms-be1098.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.

Cookbook cookbooks.sre.hosts.reimage was started by mvernon@cumin2003 for host ms-be1098.eqiad.wmnet with OS trixie

Cookbook cookbooks.sre.hosts.reimage started by mvernon@cumin2003 for host ms-be1098.eqiad.wmnet with OS trixie executed with errors:

• ms-be1098 (FAIL)
  • Removed from Puppet and PuppetDB if present and deleted any certificates
  • Removed from Debmonitor if present
  • Forced UEFI HTTP Boot for next reboot
  • Host rebooted via Redfish
  • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console ms-be1098.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.

@VRiley-WMF I tried reimaging it myself, and can confirm that it's not currently able to UEFI HTTP boot into the installer, due to the below error (which looks to be a mismatch between http vs https booting, presumably some bit of provisioning/config is incorrect?) - I've added Infrastructure-Foundations who I think will know how this should be set up...

To answer your other question: these hosts should all be JBOD, and should be installed to trixie (not bullseye - we've upgraded all the ms-backends since this procurement process started), and I'll update the ticket.

BIOS Version: 2.10.1
HTTP Boot Requested by iDRAC
Booting from HTTP Device 1: Integrated NIC 1 Port 1 Partition 1 - Broadcom Adv.
Dual 25Gb Ethernet - 40:5B:7F:81:9E:30 - IPv4
 
>>Start HTTP boot over IPv4
....
  Station IP address is 10.64.133.7
 
  Error: Access forbidden, only HTTPS connection is allowed.
 
  Error: Could not retrieve NBP file size from HTTP server.
 
  Error: Could not establish connection with HTTP server.
Boot Failed: HTTP Device 1: Integrated NIC 1 Port 1 Partition 1 - Broadcom Adv.
Dual 25Gb Ethernet - 40:5B:7F:81:9E:30 - IPv4
 
 
UEFI0417: When TLS is enabled, insecure HTTP boot without TLS is not allowed.
It is recommended to use HTTP boot over TLS for better security. To enable HTTP
over TLS, 1) Set 'TLS Authentication Mode' from 'None' to desired mode in
'Network Settings' > 'HTTP Device Settings' > 'TLS Authentication
Configuration' 2) Import appropriate certificate in 'Root Certificate
Configuration' under same page. 3) HTTP boot URI should starts with 'https://'
and configure necessary network environment. Alternatively, change 'TLS
Authentication Mode' to 'None' to disable TLS. However, please note that HTTP
without TLS is not secure.

Change #1341708 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/cookbooks@master] sre.hosts.provision: set HttpDev1TlsMode for iDRAC 9

https://gerrit.wikimedia.org/r/1341708

Cookbook cookbooks.sre.hosts.reimage was started by mvernon@cumin2003 for host ms-be1098.eqiad.wmnet with OS trixie

Cookbook cookbooks.sre.hosts.reimage started by mvernon@cumin2003 for host ms-be1098.eqiad.wmnet with OS trixie executed with errors:

• ms-be1098 (FAIL)
  • Removed from Puppet and PuppetDB if present and deleted any certificates
  • Removed from Debmonitor if present
  • Forced UEFI HTTP Boot for next reboot
  • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console ms-be1098.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.

Cookbook cookbooks.sre.hosts.reimage was started by mvernon@cumin2003 for host ms-be1098.eqiad.wmnet with OS trixie

Cookbook cookbooks.sre.hosts.reimage started by mvernon@cumin2003 for host ms-be1098.eqiad.wmnet with OS trixie executed with errors:

• ms-be1098 (FAIL)
  • Removed from Puppet and PuppetDB if present and deleted any certificates
  • Removed from Debmonitor if present
  • Forced UEFI HTTP Boot for next reboot
  • Host rebooted via Redfish
  • Host up (Debian installer)
  • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console ms-be1098.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.

@VRiley-WMF with the help of @elukey I've made some progress here. ms-be1098 now boots into the installer OK, once the disks are properly set up, it should install OK. If you can set them to all JBOD/non-RAID, then it should be good to go.

https://gerrit.wikimedia.org/r/c/operations/cookbooks/+/1341708 is the change to the provisioning cookbook that made this work - once it's merged, the other two nodes on this ticket should UEFI/HTTP boot into the installer OK.

Change #1341708 merged by Elukey:

[operations/cookbooks@master] sre.hosts.provision: set HttpDev1TlsMode for iDRAC 9

https://gerrit.wikimedia.org/r/1341708

[that change is now merged, so 1099/1100 should both provision such they can UEFT HTTP boot OK]

Cookbook cookbooks.sre.hosts.reimage was started by vriley@cumin1003 for host ms-be1098.eqiad.wmnet with OS bullseye

Cookbook cookbooks.sre.hosts.reimage was started by vriley@cumin1003 for host ms-be1098.eqiad.wmnet with OS bullseye

Cookbook cookbooks.sre.hosts.reimage started by vriley@cumin1003 for host ms-be1098.eqiad.wmnet with OS bullseye executed with errors:

• ms-be1098 (FAIL)
  • Removed from Puppet and PuppetDB if present and deleted any certificates
  • Removed from Debmonitor if present
  • Forced UEFI HTTP Boot for next reboot
  • Host rebooted via Redfish
  • Host up (Debian installer)
  • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console ms-be1098.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.

Cookbook cookbooks.sre.hosts.reimage was started by vriley@cumin2003 for host ms-be1098.eqiad.wmnet with OS trixie

Cookbook cookbooks.sre.hosts.reimage started by vriley@cumin2003 for host ms-be1098.eqiad.wmnet with OS trixie completed:

• ms-be1098 (PASS)
  • Removed from Puppet and PuppetDB if present and deleted any certificates
  • Removed from Debmonitor if present
  • Forced UEFI HTTP Boot for next reboot
  • Host rebooted via Redfish
  • Host up (Debian installer)
  • Host up (new fresh trixie OS)
  • Generated Puppet certificate
  • Signed new Puppet certificate
  • Run Puppet in NOOP mode to populate exported resources in PuppetDB
  • Found Nagios_host resource for this host in PuppetDB
  • Downtimed the new host on Icinga/Alertmanager
  • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202609162123_vriley_2604516_ms-be1098.out
  • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
  • Rebooted
  • Automatic Puppet run was successful
  • Forced a re-check of all Icinga services for the host
  • Icinga status is optimal
  • Icinga downtime removed
  • Updated Netbox data from PuppetDB
  • Updated Netbox status planned -> active
  • The sre.puppet.sync-netbox-hiera cookbook was run successfully

Cookbook cookbooks.sre.hosts.reimage was started by vriley@cumin2003 for host ms-be1099.eqiad.wmnet with OS trixie

Cookbook cookbooks.sre.hosts.reimage was started by vriley@cumin1004 for host ms-be1100.eqiad.wmnet with OS trixie