Page MenuHomePhabricator

Interactive firmware prompts on Bullseye with some Broadcom NICs
Open, MediumPublic

Description

Originally discovered as part of https://phabricator.wikimedia.org/T306148, but opening a separate task:

We have some servers with 10G Broadcom cards which emit the following message:

Some of your hardware needs non-free firmware files to operate. The firmware can be loaded from removable media, such as a USB stick or floppy.
The missing firmware files are: bnx2x/bnx2x-e2-7.13.21.0.fw bnx2x/bnx2x-e2-7.13.21.0.fw

The error message is a bit of a red herring: the tg3 driver tries to load firmware if available, but works fine without (the firmware is only needed for some additional features).

The Debian installer in bullseye introduced some changes to better deal with the fallout of missing firmware: On some GPUs people were only seeing a black screen since the firmware in question was even required to present the info screen. To mitigate that there's now code which actively detects such cases and prompts a warning. This was added in hw-detect https://tracker.debian.org/news/1245038/accepted-hw-detect-1145-source-into-unstable/ (hw-detect is a d-i component)

The case of the tg3 firmware is even mentioned there explicitly: https://www.debian.org/releases/bullseye/amd64/ch02s02 So it might be a case where the modalias information tells hw-detect that the NIC requires the firmware, while in fact is doesn't.

For bookworm this will be fixed for good by finally including the firmware in the default installation media: https://blog.einval.com/2022/04/19

I made a patch which intended to disable the firmware prompt: https://gerrit.wikimedia.org/r/784259 But sadly it doesn't appear to be working

Event Timeline

@MoritzMuehlenhoff - I've just bumped into this issue on upgrading the first prod hadoop worker and I found this bug reference, which seems highly relevant: cannot override "hw-detect/load_firmware"

The suggested solution there is to add hw-detect/load_firmware=false to the kernel command-line, since this is processed before any preseed recipes.
That would be OK for us, wouldn't it?

Would we add it in modules/install_server/files/tftpboot/bullseye-installer/pxelinux.cfg/ttyS0-115200 and modules/install_server/files/tftpboot/bullseye-installer/pxelinux.cfg/ttyS1-115200?

Change 947341 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] Preseed debian-installer not to prompt for additional firmware

https://gerrit.wikimedia.org/r/947341

Change 947341 merged by Btullis:

[operations/puppet@production] Preseed debian-installer not to prompt for additional firmware

https://gerrit.wikimedia.org/r/947341

I think that this is now fixed. My test host was an-worker1088 and it was asking for firmware for the bnx2x NIC.
After the change it went past this point in the debian-installer and installed cleanly.

Should we revert https://gerrit.wikimedia.org/r/784259 or leave it as it is?