Since we started reimaging the cp hosts to bullseye, this has come up again and I was looking at the source:

In sre/hardware/upgrade-firmware.py#L733:

self._ask_confirmation(
    f"{redfish_host.hostname}: About to reboot to apply update, please confirm"
)
if self.new:
    redfish_host.chassis_reset(ChassisResetPolicy.FORCE_RESTART)
else:
    self.spicerack.run_cookbook(
        "sre.hosts.reboot-single",
        [netbox_host.fqdn, "--reason", "bios upgrade"],
    )

Is it fine to skip this step? Put differently, I am wondering if this is equivalent to "install on next reboot" on the web management interface. If I capture the request on the web interface (NIC firmware update), I see:

<Repository><target>DCIM:INSTALLED#701__NIC.Slot.2-2-1</target><rebootType>0</rebootType></Repository>

I don't know much about this but would 0 indicate no reboot? If yes, would this then mean that if we can skip the reboot in the case of the firmware upgrade cookbook as well (assuming it is an optional argument), that this would be the same thing as "install next reboot" and thus helping solve the above?

Thanks!

@ssingh i have created a patch to defer reboots until all drivers have been uploaded. Are you able to let me know a host i can test on?

If I capture the request on the web interface ...

FYI we are using the redfish API which is slightly different to the web interface

Mentioned in SAL (#wikimedia-operations) [2023-01-26T12:41:40Z] <sukhe> depool cp3051.esams.wmnet for firmware update testing: T323717

Mentioned in SAL (#wikimedia-operations) [2023-01-26T12:42:50Z] <sukhe@cumin2002> START - Cookbook sre.hosts.downtime for 3:00:00 on cp3051.esams.wmnet with reason: T323717

Mentioned in SAL (#wikimedia-operations) [2023-01-26T12:43:05Z] <sukhe@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on cp3051.esams.wmnet with reason: T323717

In T323717#8559564, @ssingh wrote:

Since we started reimaging the cp hosts to bullseye, this has come up again and I was looking at the source:

In sre/hardware/upgrade-firmware.py#L733:

self._ask_confirmation(
    f"{redfish_host.hostname}: About to reboot to apply update, please confirm"
)
if self.new:
    redfish_host.chassis_reset(ChassisResetPolicy.FORCE_RESTART)
else:
    self.spicerack.run_cookbook(
        "sre.hosts.reboot-single",
        [netbox_host.fqdn, "--reason", "bios upgrade"],
    )

Is it fine to skip this step? Put differently, I am wondering if this is equivalent to "install on next reboot" on the web management interface. If I capture the request on the web interface (NIC firmware update), I see:

<Repository><target>DCIM:INSTALLED#701__NIC.Slot.2-2-1</target><rebootType>0</rebootType></Repository>

I don't know much about this but would 0 indicate no reboot? If yes, would this then mean that if we can skip the reboot in the case of the firmware upgrade cookbook as well (assuming it is an optional argument), that this would be the same thing as "install next reboot" and thus helping solve the above?

Thanks!

Thanks @jbond! cp3051.esams.wmnet is depooled and downtimed for three hours and ready for you. I am sure you know but as a reminder for the R440:

https://wikitech.wikimedia.org/wiki/SRE/Dc-operations/Platform-specific_documentation/Dell_Documentation#Urgent_Firmware_Revision_Notices:

Broadcom NetExtremeE firmware for 10G nic should only upgrade to 21.85.21.92, as 22.00.07.60 breaks installer.
iDrac shouldn't upgrade to 6.00.00.00 (breaks https mgmt access), cap at 5.10.30.00.

This is what I have been following for the cp hosts and it has worked so far.

If I don't upgrade the iDRAC firmware, the NIC firmware fails to update for me so I have been doing both. But IIRC, the iDRAC update didn't ask me for a reboot but I might be mistaken. (The NIC one for sure did.)

Mentioned in SAL (#wikimedia-operations) [2023-01-26T16:13:33Z] <sukhe@cumin2002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on cp3051.esams.wmnet with reason: extending downtime: T323717

Mentioned in SAL (#wikimedia-operations) [2023-01-26T16:13:49Z] <sukhe@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp3051.esams.wmnet with reason: extending downtime: T323717

In T323717#8560745, @ssingh wrote:

iDrac shouldn't upgrade to 6.00.00.00 (breaks https mgmt access), cap at 5.10.30.00.

FYI its safe to update to the most recent idrac version now, can you update where ever this information is?

In T323717#8564660, @jbond wrote:

In T323717#8560745, @ssingh wrote:

iDrac shouldn't upgrade to 6.00.00.00 (breaks https mgmt access), cap at 5.10.30.00.

FYI its safe to update to the most recent idrac version now, can you update where ever this information is?

This is from https://wikitech.wikimedia.org/wiki/SRE/Dc-operations/Platform-specific_documentation/Dell_Documentation#Urgent_Firmware_Revision_Notices: ... is it safe to do 6.x now?

Hi ssingh ,

i have just tested this by trying to upgrade the bios and the nic with only one upgrade, however im getting the following error message:

"A deployment or update operation is already in progress. Wait for the operation to conclude and then re-try."

suggesting its not possible to stack updates and they must be done one at a time

In T323717#8564666, @ssingh wrote:

In T323717#8564660, @jbond wrote:

In T323717#8560745, @ssingh wrote:

iDrac shouldn't upgrade to 6.00.00.00 (breaks https mgmt access), cap at 5.10.30.00.

FYI its safe to update to the most recent idrac version now, can you update where ever this information is?

This is from https://wikitech.wikimedia.org/wiki/SRE/Dc-operations/Platform-specific_documentation/Dell_Documentation#Urgent_Firmware_Revision_Notices: ... is it safe to do 6.x now?

Thanks have updated there and yes its safe to do 6+ now

In T323717#8564670, @jbond wrote:

Hi ssingh ,

i have just tested this by trying to upgrade the bios and the nic with only one upgrade, however im getting the following error message:

"A deployment or update operation is already in progress. Wait for the operation to conclude and then re-try."

suggesting its not possible to stack updates and they must be done one at a time

I see! Thanks for checking and looking into this. So this means that we have to reboot -- it's fine and not a big deal and certainly beats using the web interface!

Thanks @jbond for the patch and help! I can confirm that:

sudo cookbook -vvvv  -c /home/jbond/cookbook.yaml sre.hardware.upgrade-firmware "cp3051.esams.wmnet"  --no-reboot -c  nic -c idrac

Worked for cp3051: the NIC firmware installation was deferred to the next reboot and it was installed successfully, helping us avoid the double reboot.

Thanks for your help, it's much appreciated!