@RobH Why did you create a racking task only for two servers, are these shipped out in batches and we only get two initially? The order is for four servers. I'll go ahead and create the site.pp/preseed config for all four of them already anyway.

Also, I changed the names: ganeti1053/1053 were already added last year in https://phabricator.wikimedia.org/T401691.

Change #1247908 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Add ganeti1055/1056/1057/1058 to site.pp

https://gerrit.wikimedia.org/r/1247908

Change #1247908 merged by Muehlenhoff:

[operations/puppet@production] Add ganeti1055/1056/1057/1058 to site.pp

https://gerrit.wikimedia.org/r/1247908

This should indeed be for 4 servers, anyhostnames you like I made some assumptions! So this should be 4 hosts

Hey @MoritzMuehlenhoff I wanted to check something before we racked these. I'm guessing you'd like to have these spread out between cabinets, correct? I don't know if we have the space or capacity to fit all 3 in row A into a single cabinet. Let us know, thanks!

In T418903#11785174, @VRiley-WMF wrote:

I'm guessing you'd like to have these spread out between cabinets, correct? I don't know if we have the space or capacity to fit all 3 in row A into a single cabinet. Let us know, thanks!

Hi, indeed. if we can space these out across three cabinets that would be great!

These servers aren't starting out all that great. 3 of them (ganeti1055,1056 and 1057) can't power on. There are lights on the power supplies, but pushing the power button does nothing. Started with a flea power drain, then swapped out power cables. reseated power supplies, reseated the cable from the power button to the main board, and even moved ganeti1055 to and empty rack (C1) and tried to power it on there. Seems like it could be a manufactor defect? Putting in a Dell ticket for those units.

@Jhancock.wm recommended to swap the PSU's with the known working one (1058) with one of the ones that isn't working (I chose 1055)

after the full swap, it still wasn't powering on.

Ticket number for Dell is 224788872. Onsite tech will be coming onsite to check devices.

Ran provision script on 1056.

@Jclark-ctr was it able to finish the provisioning? I attempted to do this with ganeti1055, but it wasn't able to finish.

Oddly enough, the servers would only power on when the lid is removed from these units, and I'm unsure why that is.

In T418903#11802153, @VRiley-WMF wrote:

@Jclark-ctr was it able to finish the provisioning? I attempted to do this with ganeti1055, but it wasn't able to finish.

Oddly enough, the servers would only power on when the lid is removed from these units, and I'm unsure why that is.

Yes it did finish provisioning

9:45 AM <•logmsgbot> 
!log jclark@cumin1003 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1056.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED

I seem to be running into the same issue that @Jhancock.wm is running into with T418899. Awaiting to see what the fix would be.

@elukey Would you be able to maybe look into these servers as well? I feel like I'm running into the same issue that Jenn was running onto as they have the same config and same device type.

Cookbook cookbooks.sre.hosts.reimage was started by vriley@cumin1003 for host ganeti1055.eqiad.wmnet with OS bookworm

Cookbook cookbooks.sre.hosts.reimage started by vriley@cumin1003 for host ganeti1055.eqiad.wmnet with OS bookworm completed:

• ganeti1055 (PASS)
  • Removed from Puppet and PuppetDB if present and deleted any certificates
  • Removed from Debmonitor if present
  • Forced UEFI HTTP Boot for next reboot
  • Host rebooted via Redfish
  • Host up (Debian installer)
  • Host up (new fresh bookworm OS)
  • Generated Puppet certificate
  • Signed new Puppet certificate
  • Run Puppet in NOOP mode to populate exported resources in PuppetDB
  • Found Nagios_host resource for this host in PuppetDB
  • Downtimed the new host on Icinga/Alertmanager
  • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202604272054_vriley_1655260_ganeti1055.out
  • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
  • Rebooted
  • Automatic Puppet run was successful
  • Forced a re-check of all Icinga services for the host
  • Icinga status is optimal
  • Icinga downtime removed
  • Updated Netbox data from PuppetDB
  • Updated Netbox status planned -> active
  • The sre.puppet.sync-netbox-hiera cookbook was run successfully

Cookbook cookbooks.sre.hosts.reimage was started by vriley@cumin1003 for host ganeti1056.eqiad.wmnet with OS bookworm

Cookbook cookbooks.sre.hosts.reimage started by vriley@cumin1003 for host ganeti1056.eqiad.wmnet with OS bookworm completed:

• ganeti1056 (PASS)
  • Removed from Puppet and PuppetDB if present and deleted any certificates
  • Removed from Debmonitor if present
  • Forced UEFI HTTP Boot for next reboot
  • Host rebooted via Redfish
  • Host up (Debian installer)
  • Host up (new fresh bookworm OS)
  • Generated Puppet certificate
  • Signed new Puppet certificate
  • Run Puppet in NOOP mode to populate exported resources in PuppetDB
  • Found Nagios_host resource for this host in PuppetDB
  • Downtimed the new host on Icinga/Alertmanager
  • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202604272135_vriley_1699551_ganeti1056.out
  • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
  • Rebooted
  • Automatic Puppet run was successful
  • Forced a re-check of all Icinga services for the host
  • Icinga status is optimal
  • Icinga downtime removed
  • Updated Netbox data from PuppetDB
  • Updated Netbox status planned -> active
  • The sre.puppet.sync-netbox-hiera cookbook was run successfully

Cookbook cookbooks.sre.hosts.reimage was started by vriley@cumin1003 for host ganeti1057.eqiad.wmnet with OS bookworm

Cookbook cookbooks.sre.hosts.reimage started by vriley@cumin1003 for host ganeti1057.eqiad.wmnet with OS bookworm completed:

• ganeti1057 (PASS)
  • Removed from Puppet and PuppetDB if present and deleted any certificates
  • Removed from Debmonitor if present
  • Forced UEFI HTTP Boot for next reboot
  • Host rebooted via Redfish
  • Host up (Debian installer)
  • Host up (new fresh bookworm OS)
  • Generated Puppet certificate
  • Signed new Puppet certificate
  • Run Puppet in NOOP mode to populate exported resources in PuppetDB
  • Found Nagios_host resource for this host in PuppetDB
  • Downtimed the new host on Icinga/Alertmanager
  • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202604272216_vriley_1731224_ganeti1057.out
  • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
  • Rebooted
  • Automatic Puppet run was successful
  • Forced a re-check of all Icinga services for the host
  • Icinga status is optimal
  • Icinga downtime removed
  • Updated Netbox data from PuppetDB
  • Updated Netbox status planned -> active
  • The sre.puppet.sync-netbox-hiera cookbook was run successfully

Cookbook cookbooks.sre.hosts.reimage was started by vriley@cumin1003 for host ganeti1058.eqiad.wmnet with OS bookworm

Cookbook cookbooks.sre.hosts.reimage started by vriley@cumin1003 for host ganeti1058.eqiad.wmnet with OS bookworm completed:

• ganeti1058 (PASS)
  • Removed from Puppet and PuppetDB if present and deleted any certificates
  • Removed from Debmonitor if present
  • Forced UEFI HTTP Boot for next reboot
  • Host rebooted via Redfish
  • Host up (Debian installer)
  • Host up (new fresh bookworm OS)
  • Generated Puppet certificate
  • Signed new Puppet certificate
  • Run Puppet in NOOP mode to populate exported resources in PuppetDB
  • Found Nagios_host resource for this host in PuppetDB
  • Downtimed the new host on Icinga/Alertmanager
  • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202604272300_vriley_1761922_ganeti1058.out
  • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
  • Rebooted
  • Automatic Puppet run was successful
  • Forced a re-check of all Icinga services for the host
  • Icinga status is optimal
  • Icinga downtime removed
  • Updated Netbox data from PuppetDB
  • Updated Netbox status planned -> active
  • The sre.puppet.sync-netbox-hiera cookbook was run successfully

Thanks! I'll create a followup task for the cluster integration.