Change #1084253 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] search platform: add config for new search platform hosts

https://gerrit.wikimedia.org/r/1084253

Change #1084253 merged by Bking:

[operations/puppet@production] search platform: add config for new search platform hosts

https://gerrit.wikimedia.org/r/1084253

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host cloudelastic1011.eqiad.wmnet with OS bullseye

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host cloudelastic1012.eqiad.wmnet with OS bullseye

@elukey Hey luca these two are failing to provision these are custom configs

The error seems to be related to a specific network card:

PATCH https://10.65.4.200/redfish/v1/Systems/1/Bios returned HTTP 400
Response payload: {'error': {'code': 'Base.v1_10_3.GeneralError', 'message': 'A general error has occurred. See ExtendedInfo for more information.', '@Message.ExtendedInfo': [{'MessageId': 'Base.1.10.PropertyValueTypeError', 'Severity': 'Warning', 'Resolution': 'Correct the value for the property in the request body and resubmit the request if the operation failed.', 'Message': "The value 'null' for the property P1_AIOMAOC_ATGC_i2TMLAN1OPROM is of a different type than the property can accept.", 'MessageArgs': ['null', 'P1_AIOMAOC_ATGC_i2TMLAN1OPROM'], 'RelatedProperties': ['P1_AIOMAOC_ATGC_i2TMLAN1OPROM']}]}}

Note: for some reason in MessageArgs you see null but in reality we pass Legacy.

Running the cookbook with uefi makes everything working, need to figure out what is special about P1_AIOMAOC_ATGC_i2TMLAN1OPROM.

@Jclark-ctr if those are not urgent I'd ask you to leave them to me for some tests, I'll ping you when I'll find something!

I am reviewing the quote of these nodes to figure out what the item is, afaics it seems a 10G network card, but I also see a 25G capable one (with two ports, that is currently being used). Is it the case?

In T378368#10386835, @elukey wrote:

I am reviewing the quote of these nodes to figure out what the item is, afaics it seems a 10G network card, but I also see a 25G capable one (with two ports, that is currently being used). Is it the case?

From what I can see in the BIOS config we have two Intel-based-10G ports and two Supermicro-based-25G ports, and the link is present on the first of the 25G port. If I enter in any of the Intel-based-10G ports menu' I only see "UEFI Driver" "XXXX" and nothingelse related to Legacy etc.., so I suspect that these NICs don't work in a different mode.

Change #1101457 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/cookbooks@master] sre.hosts.provision: add uefi only devices for Supermicro

https://gerrit.wikimedia.org/r/1101457

Change #1101457 merged by Elukey:

[operations/cookbooks@master] sre.hosts.provision: add uefi only devices for Supermicro

https://gerrit.wikimedia.org/r/1101457

@Jclark-ctr @bking I updated the provision cookbook to support this case, but the TL;DR is that we may need to use UEFI to avoid weird configurations. In this case we can keep legacy, but I am not sure if we'll ever be able to PXE boot from one of the 10G interfaces (they are not used atm, we use the 25G ones, and we'll probably keep using those in the future, but..).

@elukey I'm fine with focusing our efforts on UEFI, it seems like the best use of our time. Ping me in IRC if I can do anything to help test.

@elukey the 10g card is copper rj45 and not in use. AOC-ATGC-i2TM. The 10g port is connected using DAC cable to AOC-A25G-b2SM.

@Jclark-ctr @bking given that the 10g card will never be used (Rj45, coppet, etc..) we can go ahead without UEFI, since that NIC will not be configured and it will not matter. The provision cookbook should run just fine, warning about the UEFI-only device for awareness. We can skip UEFI for the moment, it doesn't seem really needed, let's see how it goes.

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host cloudelastic1011.eqiad.wmnet with OS bullseye

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host cloudelastic1012.eqiad.wmnet with OS bullseye

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host cloudelastic1012.eqiad.wmnet with OS bullseye executed with errors:

• cloudelastic1012 (FAIL)
  • Removed from Puppet and PuppetDB if present and deleted any certificates
  • Removed from Debmonitor if present
  • Forced UEFI HTTP Boot for next reboot
  • Host rebooted via Redfish
  • Host up (Debian installer)
  • Add puppet_version metadata to Debian installer
  • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console cloudelastic1012.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host cloudelastic1011.eqiad.wmnet with OS bullseye executed with errors:

• cloudelastic1011 (FAIL)
  • Removed from Puppet and PuppetDB if present and deleted any certificates
  • Removed from Debmonitor if present
  • Forced UEFI HTTP Boot for next reboot
  • Host rebooted via Redfish
  • Host up (Debian installer)
  • Add puppet_version metadata to Debian installer
  • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console cloudelastic1011.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.

Change #1103367 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] WIP: Add partman recipe for raid 0 with EFI

https://gerrit.wikimedia.org/r/1103367

Change #1103381 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] cloudelastic10[12]: add hosts to new efi-based partman recipe

https://gerrit.wikimedia.org/r/1103381

Change #1103367 merged by Bking:

[operations/puppet@production] Add partman recipe for raid 0 with EFI

https://gerrit.wikimedia.org/r/1103367

Change #1103381 merged by Bking:

[operations/puppet@production] cloudelastic10[12]: add hosts to new efi-based partman recipe

https://gerrit.wikimedia.org/r/1103381

Cookbook cookbooks.sre.hosts.reimage was started by bking@cumin2002 for host cloudelastic1011.eqiad.wmnet with OS bullseye

Cookbook cookbooks.sre.hosts.reimage started by bking@cumin2002 for host cloudelastic1011.eqiad.wmnet with OS bullseye executed with errors:

• cloudelastic1011 (FAIL)
  • Removed from Puppet and PuppetDB if present and deleted any certificates
  • Removed from Debmonitor if present
  • Forced UEFI HTTP Boot for next reboot
  • Host rebooted via Redfish
  • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console cloudelastic1011.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.

Cookbook cookbooks.sre.hosts.reimage was started by bking@cumin2002 for host cloudelastic1011.eqiad.wmnet with OS bullseye

Cookbook cookbooks.sre.hosts.reimage started by bking@cumin2002 for host cloudelastic1011.eqiad.wmnet with OS bullseye executed with errors:

• cloudelastic1011 (FAIL)
  • Removed from Puppet and PuppetDB if present and deleted any certificates
  • Removed from Debmonitor if present
  • Forced UEFI HTTP Boot for next reboot
  • Host rebooted via Redfish
  • Host up (Debian installer)
  • Add puppet_version metadata to Debian installer
  • Host up (new fresh bullseye OS)
  • Generated Puppet certificate
  • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console cloudelastic1011.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.

Cookbook cookbooks.sre.hosts.reimage was started by bking@cumin2002 for host cloudelastic1012.eqiad.wmnet with OS bullseye

Cookbook cookbooks.sre.hosts.reimage started by bking@cumin2002 for host cloudelastic1012.eqiad.wmnet with OS bullseye executed with errors:

• cloudelastic1012 (FAIL)
  • Removed from Puppet and PuppetDB if present and deleted any certificates
  • Removed from Debmonitor if present
  • Forced UEFI HTTP Boot for next reboot
  • Host rebooted via Redfish
  • Host up (Debian installer)
  • Add puppet_version metadata to Debian installer
  • Host up (new fresh bullseye OS)
  • Generated Puppet certificate
  • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console cloudelastic1012.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.

Cookbook cookbooks.sre.hosts.reimage was started by bking@cumin2002 for host cloudelastic1012.eqiad.wmnet with OS bullseye

Cookbook cookbooks.sre.hosts.reimage started by bking@cumin2002 for host cloudelastic1012.eqiad.wmnet with OS bullseye completed:

• cloudelastic1012 (PASS)
  • Downtimed on Icinga/Alertmanager
  • Disabled Puppet
  • Removed from Puppet and PuppetDB if present and deleted any certificates
  • Removed from Debmonitor if present
  • Forced UEFI HTTP Boot for next reboot
  • Host rebooted via Redfish
  • Host up (Debian installer)
  • Add puppet_version metadata to Debian installer
  • Host up (new fresh bullseye OS)
  • Generated Puppet certificate
  • Signed new Puppet certificate
  • Run Puppet in NOOP mode to populate exported resources in PuppetDB
  • Found Nagios_host resource for this host in PuppetDB
  • Downtimed the new host on Icinga/Alertmanager
  • Removed previous downtime on Alertmanager (old OS)
  • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202412162119_bking_307948_cloudelastic1012.out
  • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
  • Rebooted
  • Automatic Puppet run was successful
  • Forced a re-check of all Icinga services for the host
  • Icinga status is optimal
  • Icinga downtime removed
  • Updated Netbox data from PuppetDB
  • Updated Netbox status planned -> active
  • The sre.puppet.sync-netbox-hiera cookbook was run successfully

Looks like these are ready to go into service, should they be reassigned to @bking?

(This drive-by brought to you by me looking through the WMCS procurement budget and trying to make sure everything is on track)

Change #1109483 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] cloudelastic: add cloudelastic10[12] into production

https://gerrit.wikimedia.org/r/1109483

We are currently getting a PCC error for cloudelastic1011, as its Netbox status is incorrectly set to 'planned' (should be 'active').

Mentioned in SAL (#wikimedia-operations) [2025-01-09T21:43:02Z] <bking@cumin2002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Sync cloudelastic1011 status change after Netbox update - bking@cumin2002 - T378368"

Mentioned in SAL (#wikimedia-operations) [2025-01-09T21:44:24Z] <bking@cumin2002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Sync cloudelastic1011 status change after Netbox update - bking@cumin2002 - T378368"

Mentioned in SAL (#wikimedia-operations) [2025-01-09T23:00:09Z] <inflatador> bking@puppetserver1001:~$ sudo /usr/local/sbin/puppet-facts-upload --proxy http://webproxy.eqiad.wmnet:8080 T378368

Mentioned in SAL (#wikimedia-operations) [2025-01-09T23:00:23Z] <inflatador> bking@pcc-db1002.puppet-diffs.eqiad1.wikimedia.cloud sudo -u jenkins-deploy /usr/local/sbin/pcc_facts_processor T378368

Change #1109483 merged by Bking:

[operations/puppet@production] cloudelastic: add cloudelastic10[12] into production

https://gerrit.wikimedia.org/r/1109483

Change #1110862 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] cloudelastic: remove cloudelastic100[56] from conftool, add 101[12]

https://gerrit.wikimedia.org/r/1110862