Change #1244743 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] site: add contint1003/2003 with insetup collab role

https://gerrit.wikimedia.org/r/1244743

requesting public IPs at T418520

servers racked and the idrac ip is set up. giving me a bit of trouble with the provisioning script but if you wanna steal it while i'm at lunch, i will not argue.

Change #1244743 merged by Dzahn:

[operations/puppet@production] site: add contint1003/2003 with insetup collab role

https://gerrit.wikimedia.org/r/1244743

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host contint2003.wikimedia.org with OS bookworm

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host contint2003.wikimedia.org with OS bookworm executed with errors:

• contint2003 (FAIL)
  • Removed from Puppet and PuppetDB if present and deleted any certificates
  • Removed from Debmonitor if present
  • Forced UEFI HTTP Boot for next reboot
  • Host rebooted via Redfish
  • Host up (Debian installer)
  • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console contint2003.wikimedia.org" to get a root shell, but depending on the failure this may not work.

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host contint2003.wikimedia.org with OS bookworm

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host contint2003.wikimedia.org with OS bookworm completed:

• contint2003 (PASS)
  • Removed from Puppet and PuppetDB if present and deleted any certificates
  • Removed from Debmonitor if present
  • Forced PXE for next reboot
  • Host rebooted via IPMI
  • Host up (Debian installer)
  • Checked BIOS boot parameters are back to normal
  • Host up (new fresh bookworm OS)
  • Generated Puppet certificate
  • Signed new Puppet certificate
  • Run Puppet in NOOP mode to populate exported resources in PuppetDB
  • Found Nagios_host resource for this host in PuppetDB
  • Downtimed the new host on Icinga/Alertmanager
  • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202603051659_jhancock_3916585_contint2003.out
  • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
  • Rebooted
  • Automatic Puppet run was successful
  • Forced a re-check of all Icinga services for the host
  • Icinga status is optimal
  • Icinga downtime removed
  • Updated Netbox data from PuppetDB
  • Updated Netbox status planned -> active
  • The sre.puppet.sync-netbox-hiera cookbook was run successfully

@Dzahn this is finally done. all yours!

@Jhancock.wm Thank you very much! Taking over:)