In T392851#11131759, @elukey wrote:

It seems that Virtualization cannot be disabled in the BIOS processor settings, see the Web UI:

This seems to follow something like this, that Riccardo found out (not the exact use case but it seems an example about how, in some cases, virtualization is enabled and cannot be turned off). I don't see ProcX2Apic as well in the Web UI..

Thanks for the debugging. IMO, this seems to be the sort of thing that we should follow up with Dell on, instead of you trying to figure out by yourself. If you think that makes sense, please let us know and we can take care of it.

@wiki_willy hi! We stumbled upon an issue in the Dell BIOS configs, namely it doesn't seem possible in this configuration to disable the Processor's virtualization capabilities (we do enable them only for a subset of hosts, like ganeti, but we try to disable for the rest for security reasons). Do we have anybody that we can follow up with? My last experience opening a ticket to dell wasn't great, but I can try that road if needed.

I'll open a ticket with support about this next week and followup with both support and our account team.

Thanks @RobH. Our account team has changed quite a bit, but you can follow up with Hossam and Dawn after creating the support ticket

In T392851#11132747, @RobH wrote:

I'll open a ticket with support about this next week and followup with both support and our account team.

I am testing the new provision script on other cp2xxx hosts, and I always end up with the following diff when checking if the settings have been applied:

Updated value for attribute BIOS.Setup.1-1 -> CpuInterconnectBusLinkPower (marked Set On Import to True): Disabled => Enabled
Updated value for attribute BIOS.Setup.1-1 -> EnergyPerformanceBias (marked Set On Import to True): MaxPower => BalancedPerformance
Updated value for attribute BIOS.Setup.1-1 -> PcieAspmL1 (marked Set On Import to True): Disabled => Enabled
Updated value for attribute BIOS.Setup.1-1 -> ProcC1E (marked Set On Import to True): Disabled => Enabled
Updated value for attribute BIOS.Setup.1-1 -> ProcCStates (marked Set On Import to True): Disabled => Enabled
Updated value for attribute BIOS.Setup.1-1 -> ProcPwrPerf (marked Set On Import to True): MaxPerf => OsDbpm
Updated value for attribute BIOS.Setup.1-1 -> SysProfile: PerfOptimized => PerfPerWattOptimizedOs
Updated value for attribute BIOS.Setup.1-1 -> UncoreFrequency (marked Set On Import to True): MaxUFS => DynamicUFS

This doesn't happen on cp2043, where Rob upgraded the BIOS/IDRAC/SSD firmwares. Before spending too much time in debugging this, it would be nice to upgrade the firmwares on cp2045/cp2046 and re-test the provision script. @RobH if you have time, could you please try to to the upgrades on cp2045/cp2046? I tried with the cookbook from cumin2002 but afaics the idrac is reported too old, and I am not 100% sure how to do the manual upgrade for an host when provision didn't run yet.

Summary of the status:

• Test https://gerrit.wikimedia.org/r/c/operations/cookbooks/+/1173335 to have a final version of the cookbook that runs smoothly on all new hosts. This may need upgrading the firmwares (bios/idrac/ssd) of all the hosts to get a consistent/correct behavior.
• Upgrade the bullseye installer to include a new kernel, so the SSD disks for the OS are exposed correctly in d-i.
• Wait for Dell's support to figure out why we cannot turn off CPU virtualization on single cpu hosts.

For reference please see case number below

Dear Papaul ,


Thank you for contacting Dell Technologies technical support, from this moment, I will be the point of contact in charge of assisting you in the solution of the reported problem. Below I provide you with the information corresponding to your report:

Request Number: 1228428949
Service Tag: 
Case Number: 215217522

Here is the email from Dell

Dear Papaul,

Thank you for your patience while we investigated the issue regarding the Virtualization Technology setting on your Dell PowerEdge R670.

After testing in our lab using a similar system, we confirmed that the Virtualization Technology option is enabled and greyed out by design. This behavior is consistent across all 17th generation PowerEdge servers equipped with Intel Xeon 6 processors, as documented in Set-up-BIOS-on-17th-Generation-Dell-PowerEdge-Servers (Page 16): This option is Read-only and set to Enabled on systems with Intel processors.

Unfortunately, this means the option cannot be disabled manually via BIOS. We understand this may not align with your expectations, and we recommend discussing this with your Sales Representative if this was a requirement during the procurement process.

We’re here to support you with any further technical questions or clarifications. 

Best Regards!
Jose L. Gonzalez
Enterprise Technical Support Engineer

cp2045 has had the idrac, bios, and SSD firmware updated to latest revisions to match cp2043.

Please note that these have been brought online manually one by one by @Jhancock.wm as needed, so any further updates will require some onsite work from her to bring additional units online and flashed.

Current next steps are (to my understanding):

• @elukey to test cp2045 with its latest firmware revision
  • if it works, it would be nice to determine what firmware change fixed provisioning (idrac?) as we could then manually apply that and then cookbook apply the rest
• Once fix is determined, on-sites will have to manually update the exisitng cp hosts with the firmware fix mentioned above (scope to be determined) for cookbooks to work on the rest of the hosts.
• hopefully future r470 orders have updated firmware that fixes this issue.

If we can determine exactly what firmware fixes this, then we can feedback to our Dell account team to ensure future orders are shipped with that firmware or later revisions.

In T392851#11144512, @Papaul wrote:

Here is the email from Dell

Dear Papaul,

Thank you for your patience while we investigated the issue regarding the Virtualization Technology setting on your Dell PowerEdge R670.

After testing in our lab using a similar system, we confirmed that the Virtualization Technology option is enabled and greyed out by design. This behavior is consistent across all 17th generation PowerEdge servers equipped with Intel Xeon 6 processors, as documented in Set-up-BIOS-on-17th-Generation-Dell-PowerEdge-Servers (Page 16): This option is Read-only and set to Enabled on systems with Intel processors.

Unfortunately, this means the option cannot be disabled manually via BIOS. We understand this may not align with your expectations, and we recommend discussing this with your Sales Representative if this was a requirement during the procurement process.

We’re here to support you with any further technical questions or clarifications. 

Best Regards!
Jose L. Gonzalez
Enterprise Technical Support Engineer

@MoritzMuehlenhoff : This means that we will need to enable virtualization for the cp servers as well. Historically, we haven't done that for anything other than the Ganeti boxes, so we wanted to get your input on what you think about this. Any thoughts?

In T392851#11144994, @RobH wrote:

cp2045 has had the idrac, bios, and SSD firmware updated to latest revisions to match cp2043.

Please note that these have been brought online manually one by one by @Jhancock.wm as needed, so any further updates will require some onsite work from her to bring additional units online and flashed.

Current next steps are (to my understanding):

• @elukey to test cp2045 with its latest firmware revision
  • if it works, it would be nice to determine what firmware change fixed provisioning (idrac?) as we could then manually apply that and then cookbook apply the rest
• Once fix is determined, on-sites will have to manually update the exisitng cp hosts with the firmware fix mentioned above (scope to be determined) for cookbooks to work on the rest of the hosts.
• hopefully future r470 orders have updated firmware that fixes this issue.

If we can determine exactly what firmware fixes this, then we can feedback to our Dell account team to ensure future orders are shipped with that firmware or later revisions.

Tested the cookbook on cp2045, everything went smoothly. The seems to me related to the BIOS firmware, since without it the settings indicated in T392851#11142724 don't get applied (so the cookbook keeps checking if they have been applied and fails, asking the user what to do).

@Jhancock.wm hi! Another test that we could do is to upgrade the BIOS firmware manually on cp2044, and re-test the provisioning.

@elukey it still fails with just the BIOS update. moving on to idrac and ssd updates.

@elukey

okay so what i did today in terms of firmware updates is:
cp2044 BIOS, iDRAC, SSD
cp2046 BIOS, iDRAC only
cp2047 BIOS only
please test as you see fit and I'll come back for another round tomorrow if needed.

Change #1173335 merged by Elukey:

[operations/cookbooks@master] sre.hosts.provision: update cookbook for Dell iDRAC 10

https://gerrit.wikimedia.org/r/1173335

In T392851#11149517, @Jhancock.wm wrote:

@elukey

okay so what i did today in terms of firmware updates is:
cp2044 BIOS, iDRAC, SSD
cp2046 BIOS, iDRAC only
cp2047 BIOS only
please test as you see fit and I'll come back for another round tomorrow if needed.

I've merged the new cookbook, so it is available to DCops too! I ran provisioning on all three nodes, and I got the config pushed to all of them, so I believe simply upgrading the firmware would be enough.

Small note - for cp2044 I had to run the cookbook twice, because for some reason I kept getting:

Updated value for attribute BIOS.Setup.1-1 -> SetBootOrderEn: 

RAID.SL.1-2,NIC.PxeDevice.1-1,NIC.HttpDevice.1-1 => 
RAID.SL.1-2,NIC.HttpDevice.1-1,NIC.PxeDevice.1-1

Updated value for attribute BIOS.Setup.1-1 ->  UefiBootSeq (marked Set On Import to True): 
RAID.SL.1-2, NIC.PxeDevice.1-1, NIC.HttpDevice.1-1 => 
RAID.SL.1-2, NIC.HttpDevice.1-1, NIC.PxeDevice.1-1

It was like NIC.HttpDevice.1-1 wasn't set properly, but then during the second run it worked. Still no idea why, let' see if we see this issue on the other nodes too @Jhancock.wm.

My plan is the following:

• Run test-cookbook with https://gerrit.wikimedia.org/r/c/operations/cookbooks/+/1185057 for cp2048+, to set the baseline of settings. I'll have to do it, since test-cookbook is not generally available.
• Run the upgrade firmare cookbook to upgrade idrac/bios/ssd firmwares on all hosts.
• Run the provision script (the regular one) again to apply the missing CPU settings.

Change #1185059 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/cookbooks@master] sre.hosts.provision: fix check for idrac10

https://gerrit.wikimedia.org/r/1185059

Change #1185059 merged by Elukey:

[operations/cookbooks@master] sre.hosts.provision: fix check for idrac10

https://gerrit.wikimedia.org/r/1185059

I tested the cookbook with newer nodes without the root account manually set up, and this is the result (consistent between hosts):

Updating the root user's password on the BMC.
Changing password for the account with username root: /redfish/v1/AccountService/Accounts/2
PATCH https://10.193.3.233/redfish/v1/AccountService/Accounts/2 returned HTTP 412
Response payload: {'error': {'@Message.ExtendedInfo': [{'MessageId': 'Base.1.18.PreconditionFailed', 'MessageArgs': [], 'RelatedProperties': [], 'Message': 'The ETag supplied did not match the ETag required to change this resource.', 'Severity': 'Critical', 'Resolution': 'Try the operation again using the appropriate ETag.'}], 'code': 'Base.1.18.GeneralError', 'message': 'A general error has occurred.  See Resolution for information on how to resolve the error, or @Message.ExtendedInfo if Resolution is not provided.'}}
Exception raised while executing cookbook sre.hosts.provision:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/spicerack/redfish.py", line 382, in request
    return self._api_client.request(method, uri, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/spicerack/apiclient.py", line 101, in request
    raise APIClientResponseError(response)
spicerack.apiclient.APIClientResponseError: PATCH https://10.193.3.233/redfish/v1/AccountService/Accounts/2 returned HTTP 412

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/spicerack/_menu.py", line 265, in _run
    raw_ret = runner.run()
              ^^^^^^^^^^^^
  File "/home/elukey/cookbooks_testing/cookbooks/cookbooks/sre/hosts/provision.py", line 910, in run
    self.redfish.change_user_password('root', self.mgmt_password)
  File "/usr/lib/python3/dist-packages/spicerack/redfish.py", line 549, in change_user_password
    response = self.request("patch", user_uri, json={"Password": password}, headers={"If-Match": etag})
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/spicerack/redfish.py", line 388, in request
    raise RedfishError(str(e)) from e
spicerack.redfish.RedfishError: PATCH https://10.193.3.233/redfish/v1/AccountService/Accounts/2 returned HTTP 412

With Idrac 10 there seems to be a problem when changing the root's user password via redfish, I need to investigate it :(

The change_user_password method fails because the HTTP Etag send in the HTTP PATCH call doesn't validate on the idrac side.

Something that I noticed is that the Etag returned for the root account on ms-be1081 and cp2040 (idrac 9 nodes) is something like 'ETag': '"872112-gzip"', while on idrac 10 is something like 'Etag': '"W/\'gen-3\'-gzip"'

I am wondering if the escaping plays a role, I tried different versions of it but none worked.

Found this https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/If-Match:

<etag_value>
Entity tags uniquely representing the requested resources. They are a string of ASCII characters placed between double quotes (like "675af34563dc-tr34"). They may be prefixed by W/ to indicate that they are 'weak', i.e., that they represent the resource semantically but not byte-by-byte.

However, in an If-Match header, weak entity tags will never match.

That is lovely :(

Note for self: should we remove If-Match from the change_user_password's function if the Etag is a weak validation one, or is there a better way to do it?

Change #1185877 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/software/spicerack@master] redfish: support weak Etag values in change_user_password

https://gerrit.wikimedia.org/r/1185877

This patch https://gerrit.wikimedia.org/r/c/operations/software/spicerack/+/1185877 should solve the last issue with Redfish, but it requires a spicerack release so I'll likely get to it tomorrow. After that I'll restart the provisioning work!

Change #1185877 merged by jenkins-bot:

[operations/software/spicerack@master] redfish: support weak Etag values in change_user_password

https://gerrit.wikimedia.org/r/1185877

Change #1187748 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/software/spicerack@master] redfish: increase timeout for Dell's change_user_password request

https://gerrit.wikimedia.org/r/1187748

Change #1187748 merged by Elukey:

[operations/software/spicerack@master] redfish: increase timeout for Dell's change_user_password request

https://gerrit.wikimedia.org/r/1187748

Deployed the new version of spicerack to fix the root's change user password, all good!

@Jhancock.wm Hi! I tried to run provisioning on cp2049 and it didn't find a link up, meanwhile cp2050's idrac seems not reachable from the cookbook. Are those hosts still WIP?

@elukey correct, i hadn't set those up yet. How many of the servers do you want ips setup on? I'm still trying to leave some untouched in case we need a blank slate.

In T392851#11176523, @Jhancock.wm wrote:

@elukey correct, i hadn't set those up yet. How many of the servers do you want ips setup on? I'm still trying to leave some untouched in case we need a blank slate.

@Jhancock.wm at this point we can probably set them all up, so we'll test the provision cookbook and the upgrade firmware procedures. As reminder, this is the plan:

1. Run test-cookbook with https://gerrit.wikimedia.org/r/c/operations/cookbooks/+/1185057 for cp2048 and more, to set the baseline of settings. I'll have to do it, since test-cookbook is not generally available.
2. Run the upgrade firmare cookbook to upgrade idrac/bios/ssd firmwares on all hosts. This can be done by dcops but we'll have to figure out if it works with the new hosts or not (Rob had some troubles IIRC).
3. Run the provision script (the regular one) again to apply the missing CPU settings. Dcops will run this to verify that everything works.

Lemme know!

@elukey I forgot to comment when i finished up last week. I got 2049 fixed and go 50-52 ready to go. However, we've had an issue in the new cage I wasn't able to work on 2053-2058. I will check to see if that issue is fixed as soon as i can and let you know.

@Jhancock.wm thanks! I tried 2049 today and I ended up with:

==> Unable to auto-detect NIC with link. Pick the one to set PXE on:
['NIC.Slot.5-1-1', 'NIC.Slot.5-2-1']

So it seems that there is no link up in the main NIC, is it possible?

I'd also need to ask another favor: for the remaining nodes, could you please not set up the root password? I'd need to test that calvin works fine, this is why :)

@elukey 2049 was powered off. once i powered it on the nic came up.

I'll not set the root for 2053-8

@Jhancock.wm perfect I can confirm that the provision cookbook ran fine (the test-cookbook version I mean). At this point we could use it to test if the upgrade firmware cookbook works, to then finally run again the full provision. What do you think?

yeah that's probably a good idea to do that. I finally got around to getting 53-58 iped. should be done by end of day so they're ready for you tomorrow.

update. everything but 2056 is ready. that one has a physical issue. the console and idrac connections are on a removeable card on these new models and the idrac port isn't responding to anything. I'm gonna open up a support ticket with dell for that and see if i can get a fix or a replacement for it. (fwiw, the ip is configured, but i can't get it to ping no matter what i do or double check.)

For cp2050 I keep getting this:

GET https://10.193.3.234/redfish/v1/TaskService/TaskMonitors/JID_580944559377 returned HTTP 400
Response payload: {'error': {'@Message.ExtendedInfo': [{'MessageId': 'IDRAC.2.11.SYS051', 'MessageArgs': [], 'RelatedProperties': [], 'Message': 'The system could not be shut down within the specified time.', 'Severity': '', 'Resolution': '', 'Oem': {}}], 'code': 'Base.1.18.GeneralError', 'message': 'A general error has occurred.  See Resolution for information on how to resolve the error, or @Message.ExtendedInfo if Resolution is not provided.'}}

Then the provision cookbook is not able to finish, even if I retry a lot of times. No idea why, but I am going to proceed with the rest and see if we find the same pattern.

cp2051 worked, cp2052 showed the issue, cp2053 worked.

Done up to cp2058, all good (excluding cp2056 as requested).

Next steps:

• Upgrade firmwares
• Check why the cookbook didn't run on cp2052 and cp2050 :(

@elukey heads up, i'm gonna try swapping the console card with another CP server to see if it's the card or something else. will probably be 2058.

@Jhancock.wm I tried to use the firmware upgrade cookbook but it bails out due to this:

cp2048: SKIPPING - iDRAC version (1.20.25.0) is too low to perform updates.  please upgrade iDRAC to version 3.30.30.30 before proceeding

So I have the feeling that idrac/ssd/bios firmware upgrades will need to be done manually :( :(

since everything is reachable via idrac/mgmt now, i should be able to tackle that as a background task. I'll see how many i can get done tonight and let you know.

In T392851#11193969, @Jhancock.wm wrote:

since everything is reachable via idrac/mgmt now, i should be able to tackle that as a background task. I'll see how many i can get done tonight and let you know.

@Jhancock.wm there may be a problem in the firmware cookbook, I'll try to work on it today/tomorrow, something is off with the reported firmware version of IDRAC 10+. If it is not a problem could you please leave cp2048 aside for my tests? Also it is fine if you want to upgrade next week, by that time I hope I'll have either a better cookbook or a more definitive "can't do it" answer :(

@elukey I can wait! wasn't trying to rush you. lemme know next week and we'll take care of it then. =)

Change #1189518 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/software/spicerack@master] redfish: improve log_entries for idrac 10

https://gerrit.wikimedia.org/r/1189518

In T392851#11194034, @Jhancock.wm wrote:

@elukey I can wait! wasn't trying to rush you. lemme know next week and we'll take care of it then. =)

@Jhancock.wm Hi! I am working on a solution with Jesse about some new corner cases with Idrac 10 (that are causing the firmware cookbook to fail). We'll have something to test later on during the week, so lemme me know what you want to do. If you want to wait we can test maximum next week how the cookbook works, possibly avoiding a ton of manual updates for you :(

@elukey i'm okay with waiting.

Change #1189518 merged by Elukey:

[operations/software/spicerack@master] redfish: improve log_entries for idrac 10

https://gerrit.wikimedia.org/r/1189518

Update on the cp2056. Finally got Dell to agree to send a replacement card after a week of back of forth and escalations. So that should be fixed no later than Wednesday.

The firmware cookbook doesn't work yet since spicerack is configured to look for a HttpPushUri field in the Redfish's UpdateService endpoint, but iDRAC 10 now only offers:

'MultipartHttpPushUri': '/redfish/v1/UpdateService/MultipartUpload',

cp2056 has had the card replaced and i've assigned the mgmt ip to it. if you needed a clean slate for any reason it's ready.

I managed to have an idrac upgrade triggered by the cookbook, but it then failed when checking the state of the idrac (that was down because of the maintenance in progress):

2025-10-01 14:24:29,448 elukey 446619 [WARNING] [10/30, retrying in 30.00s] Polling task: JID_593283923873 not completed yet: status=OK, state=Running, completed=25%
2025-10-01 14:24:59,932 elukey 446619 [INFO] [IDRAC.2.11.PR20] Job in progress.
2025-10-01 14:24:59,933 elukey 446619 [WARNING] [11/30, retrying in 30.00s] Polling task: JID_593283923873 not completed yet: status=OK, state=Running, completed=95%
2025-10-01 14:25:29,936 elukey 446619 [WARNING] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7ff1dc793290>: Failed to establish a new connection: [Errno
 111] Connection refused')': /redfish/v1/TaskService/TaskMonitors/JID_593283923873
2025-10-01 14:25:31,938 elukey 446619 [WARNING] Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7ff1dc76a410>: Failed to establish a new connection: [Errno
 111] Connection refused')': /redfish/v1/TaskService/TaskMonitors/JID_593283923873

[..]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/spicerack/apiclient.py", line 91, in request
    response = self._http_session.request(method, url, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 587, in request
    resp = self.send(prep, **send_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 701, in send
    r = adapter.send(request, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/wmflib/requests.py", line 65, in send
    return super().send(request, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/requests/adapters.py", line 556, in send
    raise RetryError(e, request=request)
requests.exceptions.RetryError: HTTPSConnectionPool(host='10.193.2.183', port=443): Max retries exceeded with url: /redfish/v1/TaskService/TaskMonitors/JID_593283923873 (Caused by ResponseError('too many 503 error responses'))

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/spicerack/redfish.py", line 382, in request
    return self._api_client.request(method, uri, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/spicerack/apiclient.py", line 98, in request
    raise APIClientError(message) from e
spicerack.apiclient.APIClientError: Failed to perform GET request to https://10.193.2.183/redfish/v1/TaskService/TaskMonitors/JID_593283923873

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/spicerack/_menu.py", line 265, in _run
    raw_ret = runner.run()
              ^^^^^^^^^^^^
  File "/home/elukey/cookbooks_testing/cookbooks/cookbooks/sre/hardware/upgrade-firmware.py", line 1072, in run
    failures += self._run_host(hostname)
                ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/elukey/cookbooks_testing/cookbooks/cookbooks/sre/hardware/upgrade-firmware.py", line 1102, in _run_host
    self.update_idrac(redfish_host, netbox_host)
  File "/home/elukey/cookbooks_testing/cookbooks/cookbooks/sre/hardware/upgrade-firmware.py", line 719, in update_idrac
    self.poll_id(redfish_host, job_id, True)
  File "/home/elukey/cookbooks_testing/cookbooks/cookbooks/sre/hardware/upgrade-firmware.py", line 500, in poll_id
    return redfish_host.poll_task(job_id)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/wmflib/decorators.py", line 231, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/spicerack/redfish.py", line 481, in poll_task
    response = self.request("get", uri)
               ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/spicerack/redfish.py", line 390, in request
    raise RedfishError(str(e)) from e

My understanding is that the poll job-id fails since the idrac reboots after the firmware is installed, loosing connectivity.

Change #1193046 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/software/spicerack@master] redfish: allow HTTP 204 responses in poll_task

https://gerrit.wikimedia.org/r/1193046

Change #1193046 merged by Elukey:

[operations/software/spicerack@master] redfish: allow HTTP 204 responses in poll_task

https://gerrit.wikimedia.org/r/1193046

Change #1192898 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/cookbooks@master] sre.hardware.upgrade-firmware: add support for IDRAC 10

https://gerrit.wikimedia.org/r/1192898

Status update: I was able to upgrade idrac+bios of most of the cp hosts, I'll review the remaining ones on Monday and I'll give a precise list in here if there will be outstanding problems or not.

The code to make the firmware upgrade cookbook to work is still under review, but I tested it and it works :)

In T392851#11242196, @elukey wrote:

Status update: I was able to upgrade idrac+bios of most of the cp hosts, I'll review the remaining ones on Monday and I'll give a precise list in here if there will be outstanding problems or not.

The code to make the firmware upgrade cookbook to work is still under review, but I tested it and it works :)

Thank you so much for your and Jenn's work on this, on behalf of Traffic!

Change #1192898 merged by Elukey:

[operations/cookbooks@master] sre.hardware.upgrade-firmware: add support for IDRAC 10

https://gerrit.wikimedia.org/r/1192898

Change #1193818 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/cookbooks@master] sre.hardware.upgrade-firmware: fix ssd/storage corner cases

https://gerrit.wikimedia.org/r/1193818

I was able to update the firmware cookbook for IDRAC 10, and now we can do idrac+bios (still working on some issue with ssd, should be solved soon). I rolled out the idrac+bios upgrades to all the nodes, these are the problematic ones:

• cp2050: couldn't run the upgrade firmware, I tried the provision one but it returned me a lot of HTTP 400 and I can't explain why. Maybe it is in a weird state and it needs a BMC reboot?
• cp2052: same as above, need more follow up.
• cp2056: I tried provisioning but ended up with:

Response payload: {'error': {'@Message.ExtendedInfo': [{'Message': 'A required license is missing or expired.', 'MessageArgs': [], 'MessageArgs@odata.count': 0, 'MessageId': 'IDRAC.2.11.LIC501', 'RelatedProperties': [], 'RelatedProperties@odata.count': 0, 'Resolution': 'Obtain an appropriate license from Dell Digital Locker and then try again. If the problem persists, refer to the product documentation or contact technical support.', 'Severity': 'Warning'}], 'code': 'Base.1.18.GeneralError', 'message': 'A general error has occurred. See ExtendedInfo for more information'}}

@Jhancock.wm If you have time could you please check the above hosts? I'd really like your option, not sure what's wrong :)

In the meantime, I'll work on upgrading the ssd fimrwares!

cp2050/52: try logging into root and using the racadm command. That's one of the symptoms when it needs to be used.
cp2056: this is the one where i had to replace the card that housed the idrac. I bet something didn't transfer properly. I'll login to it and see if i can find a way to get that updated.

@elukey give 2056 a shot when you get on next. might have fixed it. if not i might have to get dell involved again.

@Jhancock.wm tried again, then reset the idrac on 2056, re-run again but same error :(

I've reset the IDRAC for cp2052 and I was able to upgrade the firmwares! \o/

No luck with cp2050, provision ends up in HTTP 400s and I cannot connect to the BMC to reset it etc.. I think it needs some manual actions from the dcops side (like configuring network, reset the idrac etc..)

@elukey fixed cp2050. opening a ticket for cp2056

I was able to provision and upgrade idrac+bios on 2050, thanks!

All cp hosts (but 2056) have the latest bios+idrac and I've run the complete version of the provision cookbook to apply the whole set of BIOS settings.

Next steps:

• Keep working on 2056 (Dcops cut a ticket for Dell etc..).
• Figure out why PXE booting doesn't work for cp2043
• Apply the latest SSD firmware to all hosts (not strictly necessary but it is best to keep things consistent to avoid surprises in the future). This depends on a change for the upgrade-firmware cookbook that I am working on.
• Complete the work on T405102 to allow Bullseye on these nodes.
• Reimage all hosts.

Change #1193818 merged by Elukey:

[operations/cookbooks@master] sre.hardware.upgrade-firmware: fix ssd upgrade

https://gerrit.wikimedia.org/r/1193818

Change #1194883 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/cookbooks@master] sre.hardware.upgrade-firmware: use lower when matching firmware versions

https://gerrit.wikimedia.org/r/1194883

Change #1194883 merged by Elukey:

[operations/cookbooks@master] sre.hardware.upgrade-firmware: use lower when matching firmware versions

https://gerrit.wikimedia.org/r/1194883

@elukey license uploaded for cp2056. should be good to try that one again.

Change #1194969 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/cookbooks@master] sre.hardware.upgrade-firmware: improve matching for SSD checks

https://gerrit.wikimedia.org/r/1194969

In T392851#11261019, @Jhancock.wm wrote:

@elukey license uploaded for cp2056. should be good to try that one again.

Host is good now! Thanks!

Cross-posting the comment from T405102#11273708,

Traffic discussed this in the team meeting today. We decided that given the above blocker, we should simply move to trixie and use OpenSSL (3.5.0) as shipped by trixie. The reasons for doing so are this: it doesn't make sense to upgrade to bookworm as that ships OpenSSL 3.0 and we have concerns around the performance (we haven't evaluated that as we have with 3.5 but we believe that to be the case from what we have seen).
It also does not make sense to spend time and resources upgrading to just bookworm when trixie has been released. Given that, we will spend our efforts in preparing the cp hosts for trixie upgrade, with a planned rollout for Q3 (Jan).
This means that for this period -- until possibly February 2026 -- the cp hosts in codfw will not be provisioned to serve live traffic. We are hesitant in rolling out trixie to the cp hosts before the December break, given that it will take us some time to prepare and test the trixie upgrade. If we can get the upgrade up by that time, we will work on it, otherwise this is a stretch goal.

SSD firmwares updated on all cp hosts! So at this point we can try to reimage all hosts to trixie.

For some reason cp2043 wasn't able to PXE boot the last time that I tried, will report if the issue persists.

@Jhancock.wm Hi! So I've reimaged cp2044 with Debian Trixie and everything went fine, we can proceed to reimage the rest with Trixie and see how it goes. For some reason cp2043 seems in a weird state, could you please check? I tried to connect to the WebUI but it doesn't work, and the hosts doesn't PXE boot.

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin1002 for host cp2045.codfw.wmnet with OS bullseye

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin1002 for host cp2045.codfw.wmnet with OS bullseye executed with errors:

• cp2045 (FAIL)
  • Removed from Puppet and PuppetDB if present and deleted any certificates
  • Removed from Debmonitor if present
  • Forced UEFI HTTP Boot for next reboot
  • Host rebooted via Redfish
  • Host up (Debian installer)
  • Add puppet_version metadata (7) to Debian installer
  • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console cp2045.codfw.wmnet" to get a root shell, but depending on the failure this may not work.

@elukey hey how do i reimage with Debian Trixie. That seems different than running the reimage cookbook.

@Jhancock.wm --os trixie is sufficient. Did you encounter any issue while doing 2045?

FWIW doing one or two hosts is more than enough. We will reimage them again anyway so it doesn't make sense IMO for you both to spend time upgrading all of them to trixie. If one or two reimage fine, please leave the rest to us.

@ssingh 2043 and 2044 have been reimaged. so it's all yours!

@elukey i spaced we have a new os lol. i tried to do bullseye per the original instructions. I was able to confirm that 2043 was reimaged and loads into the os login screen. (if you're using firefox to login to the webui, sometimes it hangs and you gotta restart the broswer)

Change #1194969 merged by JHathaway:

[operations/cookbooks@master] sre.hardware.upgrade-firmware: improve matching for SSD checks

https://gerrit.wikimedia.org/r/1194969

The IDRAC 10 support for provision + upgrade-firmware is done, we should be good from the I/F and dcops point of view :)

Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin1003 for host cp2043.codfw.wmnet with OS trixie

Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin1003 for host cp2043.codfw.wmnet with OS trixie executed with errors:

• cp2043 (FAIL)
  • Removed from Puppet and PuppetDB if present and deleted any certificates
  • Removed from Debmonitor if present
  • Forced UEFI HTTP Boot for next reboot
  • Host rebooted via Redfish
  • Host up (Debian installer)
  • Add puppet_version metadata (7) to Debian installer
  • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console cp2043.codfw.wmnet" to get a root shell, but depending on the failure this may not work.

@ssingh do you need assistance getting these reimaged?

In T392851#11514052, @Jhancock.wm wrote:

@ssingh do you need assistance getting these reimaged?

Thanks for the offer, @Jhancock.wm. We usually do these ourselves, but we will certainly ask for help if required. Thank you!