Allow UEFI DHCP configs
Open, LowPublic
Actions

Assigned To

None

Authored By

	elukey
	Aug 28 2024, 1:05 PM

Description

It seems a good time to test if UEFI could be used in our infrastructure. The rationale is that more and more issues arose when dealing with Legacy BIOS / lpxelinux / TFTP / etc.. during the past years (see T363576 for some details). Most of the vendors asks to us if we have tested it with UEFI, since Legacy BIOS is not really fully supported anymore.

In order to be able to use UEFI, we should:

Add syslinux-efi to our DHCP install servers.
Verify if anything needs to be added to Spicerack's dhcp module to allow UEFI. In theory it should be a matter of configuring filename and some option pxelinux.something, that is already possible.
Configure the provision cookbook to set UEFI, istead of Legacy BIOS (opt-in with a parameter).
Configure the reimage cookbook to use UEFI as well (same opt-in parameter).

Once we have a running host with UEFI that works with provision and reimage we can think about next steps.

Everything is of course not taking care of the security review that will be needed if we choose UEFI, that should be on a separate task in my opinion.

Details

Related Changes in Gerrit:

Subject	Repo	Branch	Lines +/-
wdqs1025: Configure partitions for UEFI	operations/puppet	production	+4 -1
partman: add recipe for UEFI 4-disk SW RAID-10	operations/puppet	production	+15 -0
sre.hosts.reimage: fix remote command to use to test if d-i started	operations/cookbooks	master	+2 -1
sre.hosts.reimage: fix _validate() when using UEFI	operations/cookbooks	master	+4 -1
sre.hosts.reimage: add UEFI HTTP Boot support	operations/cookbooks	master	+63 -22
sre.hosts.provision: initial UEFI support	operations/cookbooks	master	+57 -18
efi: add efi boot files on apt server	operations/puppet	production	+49 -0

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Open	None	T373519 Allow UEFI DHCP configs
Open	None	T376944 Update partman configs to allow for UEFI or BIOS booting
Open	None	T376949 UEFI and software RAID
Open	None	T381919 Supermicro: unable to set boot order after using Redfish to boot once
Open	jhathaway	T383173 Supermicro: UEFI HTTP boot request hangs on cold boot
Resolved	jhathaway	T394847 Supermicro: test if Intel card exhibits the same cold boot behavior
Resolved	Jhancock.wm	T400211 Install serial port breakout card on sretest2001
		Unknown Object (Task)

Event Timeline

elukey created this task.Aug 28 2024, 1:05 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 28 2024, 1:05 PM

MoritzMuehlenhoff subscribed.Sep 2 2024, 2:29 PM

cmooney subscribed.Sep 2 2024, 2:43 PM

• joanna_borun triaged this task as Low priority.Sep 2 2024, 2:44 PM

jhathaway subscribed.Sep 4 2024, 2:00 PM

Add syslinux-efi to our DHCP install servers.

Syslinux development seems to have halted upstream in 2019, should we look consider using another bootloader for EFI?

In T373519#10127332, @jhathaway wrote:

Add syslinux-efi to our DHCP install servers.

Syslinux development seems to have halted upstream in 2019, should we look consider using another bootloader for EFI?

@jhathaway definitely, I thought it could have been a good start even for quick tests, but we can test something else too. We can customize what DHCP sends to the NIC doing PXE via spicerack/cookbook, so we could have multiple boot loaders for EFI installed and test them separately. Do you have any suggestion about others to pick up or test?

In spicerack we now have dhcp_options and dhcp_filename to pass to DHCPConfMac and DHCPConfOpt82. This allows us to override filename and various option $something settings in cookbooks.

Something like:

filename "efi64/syslinux.efi";
option pxelinux.pathprefix "http://XXX.XXX.XXX.XXX/efi64/";

is possible just passing the right params when creating the class. It could be used in the provision cookbook when --force-uefi is set, for example.

Change #1077377 had a related patch set uploaded (by Ayounsi; author: Ayounsi):

[operations/cookbooks@master] sre.hosts.provision: initial UEFI support

https://gerrit.wikimedia.org/r/1077377

gerritbot added a project: Patch-For-Review.Oct 2 2024, 1:09 PM

Change #1077497 had a related patch set uploaded (by JHathaway; author: JHathaway):

[operations/cookbooks@master] sre.hosts.reimage: add UEFI HTTP Boot support

https://gerrit.wikimedia.org/r/1077497

Change #1078020 had a related patch set uploaded (by JHathaway; author: JHathaway):

[operations/puppet@production] efi: add efi boot files on apt server

https://gerrit.wikimedia.org/r/1078020

Change #1078020 merged by JHathaway:

[operations/puppet@production] efi: add efi boot files on apt server

https://gerrit.wikimedia.org/r/1078020

ayounsi subscribed.Oct 14 2024, 3:08 PM

bking subscribed.Oct 31 2024, 2:44 PM

Forgive the drive-by comment, but I'm wondering if we have evaluated any other NICs besides Broadcom? We've lost countless hours to their firmware bugs (at least ~100 of my team's hosts have been affected in the ~3 years I've worked here). That's a pretty significant cost if you think about our salaries, opportunity costs, etc.

Broadcom had a very low reputation at previous places I've worked, so I thought it might be helpful to evaluate Intel (or any other brand) of NIC to see what's a problem with legacy BIOS vs what's a problem with Broadcom. Apologies if this has already been considered.

@bking I think it's a question worth asking, but probably not in that task :) Could you open a dedicated one for the Procurement/DCops team?

MatthewVernon mentioned this in T378584: Evaluate hw-raid controllers for Supermicro's Config J.Nov 1 2024, 11:51 AM

bking mentioned this in T378824: Evaluate alternatives to Broadcom NICs.Nov 1 2024, 4:42 PM

Change #1077377 merged by Elukey:

[operations/cookbooks@master] sre.hosts.provision: initial UEFI support

https://gerrit.wikimedia.org/r/1077377

elukey mentioned this in T371400: Q1:rack/setup/install ms-be208[1-8].Nov 4 2024, 2:44 PM

Change #1077497 merged by JHathaway:

[operations/cookbooks@master] sre.hosts.reimage: add UEFI HTTP Boot support

https://gerrit.wikimedia.org/r/1077497

Maintenance_bot removed a project: Patch-For-Review.Nov 4 2024, 10:31 PM

BTullis subscribed.Nov 5 2024, 1:42 PM

Change #1087865 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/cookbooks@master] sre.hosts.reimage: fix _validate() when using UEFI

https://gerrit.wikimedia.org/r/1087865

gerritbot added a project: Patch-For-Review.Nov 6 2024, 9:09 AM

Change #1087865 merged by Elukey:

[operations/cookbooks@master] sre.hosts.reimage: fix _validate() when using UEFI

https://gerrit.wikimedia.org/r/1087865

I found two issues while reimaging ms-be2083 (supermicro):

The cookbook can't recognize if it is in d-i or not, since /proc/cmdline doesn't contain "debian-installer" as expected but:

~ # cat /proc/cmdline 
linux initrd=one.gz vga=normal auto-install/enable=true preseed/url=http://apt.wikimedia.org/autoinstall/preseed.cfg DEBCONF_DEBUG=5 netcfg/choose_interface=auto netcfg/get_hostname=unassigned netcfg/get_domain=unassigned netcfg/dhcp_timeout=60 --- console=ttyS1,115200n8 raid0.default_layout=2

The recipe for ms-be2083 may be wrong since during boot it can't find any media present, and forces PXE again (that triggers d-i etc...).

Maintenance_bot removed a project: Patch-For-Review.Nov 6 2024, 9:31 AM

Change #1087869 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/cookbooks@master] sre.hosts.reimage: fix remote command to use to test if d-i started

https://gerrit.wikimedia.org/r/1087869

gerritbot added a project: Patch-For-Review.Nov 6 2024, 9:37 AM

Change #1087869 merged by Elukey:

[operations/cookbooks@master] sre.hosts.reimage: fix remote command to use to test if d-i started

https://gerrit.wikimedia.org/r/1087869

In T373519#10295331, @elukey wrote:

The recipe for ms-be2083 may be wrong since during boot it can't find any media present, and forces PXE again (that triggers d-i etc...).

No idea what happened but I retried a reimage after https://gerrit.wikimedia.org/r/1087869 and everything went fine. So for the moment the first real test with ms-be2083 seemed a success! We'll see what DP want to do next, but we'll surely have more hosts to battle test the new UEFI support.

Maintenance_bot removed a project: Patch-For-Review.Nov 6 2024, 10:30 AM

Change #1099740 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] partman: add recipe for UEFI 4-disk SW RAID-10

https://gerrit.wikimedia.org/r/1099740

gerritbot added a project: Patch-For-Review.Dec 2 2024, 4:27 PM

Change #1099740 merged by Bking:

[operations/puppet@production] partman: add recipe for UEFI 4-disk SW RAID-10

https://gerrit.wikimedia.org/r/1099740

Change #1101095 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] wdqs1025: Configure partitions for UEFI

https://gerrit.wikimedia.org/r/1101095

Change #1101095 merged by Bking: