@cmooney thanks for the write up, i attached on older task to this which list some of the edge cases and issues etc. overall i think the the proposal is good. as to the open question of how to get the interfaces names. We may be able to add this to the provisioning script. with our configuration we are using "Predictable Interface Names" which are based on the device location i.e. bus, slot etc and not the driver. We may be able to use redfish to get this information (although i couldn't find it from a quick look) and the update netbox. We have also spoken in the past of having some very simple debian live system that we could boot into during provisioning to grab a bunch of data from the host and update netbox.

In T347411#9200644, @jbond wrote:

We may be able to use redfish to get this information (although i couldn't find it from a quick look) and the update netbox.

Hmm that's a very interesting prospect might well be a way to do it.

We have also spoken in the past of having some very simple debian live system that we could boot into during provisioning to grab a bunch of data from the host and update netbox.

Another good suggestion - thanks!

It's great to see momentum on this recurring pain point!

To add to it, we could have the hosts boot up with only a v6 SLAAC IP (decommission the DHCP) and then get their final/permanent IP at their first Puppet run.

To update Netbox with the proper list of interfaces, this could be done at the early or late_command stage (cf. https://github.com/wikimedia/operations-puppet/tree/production/modules/install_server/files/autoinstall/scripts )
Or even sooner if we use something like iPXE (which could also workaround the issue we're having in T304483: PXE boot NIC firmware regression

Next step should probably to have a meeting about that, then open (or update) sub tasks with the direction we want to take.

I had a little play with the redfish api and the PCIe info is available. Unfortunately Linux interfaces remain about as predictable as the Lotto numbers.

Taking a random system, cloudservices1006, it has interfaces as follows:

cmooney@cloudservices1006:~$ lspci | grep -i ethernet
04:00.0 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5720 2-port Gigabit Ethernet PCIe
04:00.1 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5720 2-port Gigabit Ethernet PCIe
4b:00.0 Ethernet controller: Broadcom Inc. and subsidiaries BCM57412 NetXtreme-E 10Gb RDMA Ethernet Controller (rev 01)
4b:00.1 Ethernet controller: Broadcom Inc. and subsidiaries BCM57412 NetXtreme-E 10Gb RDMA Ethernet Controller (rev 01)

From these, systemd-udevd.service creates the following udev properties for ID_NET_NAME_PATH (described by debian here and systemd here):

cmooney@cloudservices1006:~$ udevadm info -e | grep ID_NET_NAME_PATH
E: ID_NET_NAME_PATH=enp4s0f0
E: ID_NET_NAME_PATH=enp4s0f1
E: ID_NET_NAME_PATH=enp75s0f0np0
E: ID_NET_NAME_PATH=enp75s0f1np1

Using the redfish API we can pull the PCI IDs (ultimately the same as shown in lspci although they are decimal not hex here), and almost predict the names:

cmooney@cumin1001:~$ ./get_nic_pci.py --host cloudservices1006.mgmt.eqiad.wmnet
iDRAC Password: 
NIC.Embedded.1-1-1 - enp4s0f0 - NetXtreme BCM5720 Gigabit Ethernet PCIe
NIC.Embedded.2-1-1 - enp4s0f1 - NetXtreme BCM5720 Gigabit Ethernet PCIe
NIC.Integrated.1-1-1 - enp75s0f0 - BCM57412 NetXtreme-E 10Gb RDMA Ethernet Controller
NIC.Integrated.1-2-1 - enp75s0f1 - BCM57412 NetXtreme-E 10Gb RDMA Ethernet Controller

Comparing that to what the system shows the only difference is the "np0" and "np1" suffix that get assigned to the 10G ("integrated") ones rather than the 1G ("embedded") ones. In all cases the digit after the "np" matches the PCIe function number, but I can't quite see why it is adding them in one case and not the other. Looking at the full output of the API for these devices, and indeed querying their udev properties, I can't see what difference is present to result in one having the 'npX' and one not.

There is possibly a clue in the ID_NET_LABEL_ONBOARD, which in the free-text shows separate 'NICs' for the embedded, but a single NIC with two ports for the integrated:

cmooney@cloudservices1006:~$ udevadm info -e | grep ID_NET_LABEL_ONBOARD
E: ID_NET_LABEL_ONBOARD=Embedded NIC 1
E: ID_NET_LABEL_ONBOARD=Embedded NIC 2
E: ID_NET_LABEL_ONBOARD=Integrated NIC 1 Port 1-1
E: ID_NET_LABEL_ONBOARD=Integrated NIC 1 Port 2-1

That all may be academic, however, as the netdev names are not being set based on PCI location. Instead they're named as follows:

cmooney@cloudservices1006:~$ ip -br link show | grep ^en
eno12399np0      UP             00:62:0b:c9:19:70 <BROADCAST,MULTICAST,UP,LOWER_UP> 
eno8303          DOWN           c4:5a:b1:ab:d7:f8 <BROADCAST,MULTICAST> 
eno12409np1      DOWN           00:62:0b:c9:19:71 <BROADCAST,MULTICAST> 
eno8403          DOWN           c4:5a:b1:ab:d7:f9 <BROADCAST,MULTICAST>

These names are because the NICs are exposed as "onboard" devices, which means ID_NET_NAME_ONBOARD is populated, and as per the default NamePolicy that is preferred. Ultimately those numbers come from the device's ACPI Index ID:

cmooney@cloudservices1006:~$ cat /sys/class/net/eno8303/device/acpi_index 
8303
cmooney@cloudservices1006:~$ cat /sys/class/net/eno12399np0/device/acpi_index 
12399

Certain NICs in our estate are not seen as 'onboard', and expose no 'acpi index'. This results in no ID_NET_NAME_ONBOARD being populated for them, in which case the system does use ID_NET_NAME_PATH for the name:

cmooney@dns5003:~$ ip -br addr show  | egrep ^en
enp94s0f0np0     UP             103.102.166.10/28 2001:df2:e500:1:103:102:166:10/64 fe80::8616:cff:fe5d:a880/64 
eno1             DOWN           
enp94s0f1np1     DOWN           
eno2             DOWN

Either way, despite having a good look, I couldn't find any way to get these ACPI Index IDs from the redfish API. So this may be a bit of a dead-end.

Some things that may be possible, if still trying to predict the names from redfish data:

1. Change the NamePolicy we configure to "path", so systemd will use the name from ID_NET_NAME_PATH (derived from pci location), rather than the 'onboard'/acpi ID.
   • We still need to work out when and why it adds 'npX' suffix to do this
2. Change the NamePolicy to 'mac', so we get names like enxc45ab1abd7f5. Ugly, but should be consistent?

The idea of booting the system at the provision stage into some "simple live system" and grabbing the information is also a good option perhaps.

In T347411#9203208, @ayounsi wrote:

To add to it, we could have the hosts boot up with only a v6 SLAAC IP (decommission the DHCP) and then get their final/permanent IP at their first Puppet run.

Definitely a good option. I think we need to be mindful of T102099: Fix IPv6 autoconf issues once and for all, across the fleet. here (esp. the weird bug documented in T102099#8537693). The idea of disabling RAs and SLAAC completely did appeal to me, but no srong feelings. I guess in this case we can pull the system's MAC from redfish API (or maybe switch?) and then derive the SLAAC IP?

To update Netbox with the proper list of interfaces, this could be done at the early or late_command stage (cf. https://github.com/wikimedia/operations-puppet/tree/production/modules/install_server/files/autoinstall/scripts )

That's a very good point. There is no hard requirement to update netbox with the real names at 'provision' stage, the only requirement is that they are available before we try to build the network conf files. If we can control the order of operations and have an 'export' function be run by puppet before it tries to build the config it might be an easy win.

Next step should probably to have a meeting about that, then open (or update) sub tasks with the direction we want to take.

Agreed I think that makes sense. We can possibly discuss briefly during our quarterly planning and if there is agreement to proceed schedule a dedicated one?

In T347411#9203210, @cmooney wrote:

Some things that may be possible, if still trying to predict the names from redfish data:

1. Change the NamePolicy to 'mac', so we get names like enxc45ab1abd7f5. Ugly, but should be consistent?

Another idea similar to this, suggested by @faidon (thanks!), would be to push a custom /etc/udev/rules.d/70-persistent-net.rules file to our servers. We could define rules in this to set a custom name for interfaces based on the device's MAC address (I think similar to answer 2 here).

So potentially we could use something generic like "eth0" in Netbox for the interface name at provision stage, then in first puppet run we could add the rules to give that name to the primary interface, and write a new /etc/network/interfaces with that name and the full config (incl. vlan ints, bridges etc). Lots to be worked out of course, but it might be a way to use the MAC address to make things predictable, without needing the MAC at provision stage in netbox.

Faidon also pointed me to a systemd bug he raised a while ago about a very similar issue:

https://github.com/systemd/systemd/issues/12261

Certain NICs in our estate are not seen as 'onboard', and expose no 'acpi index'. This results in no ID_NET_NAME_ONBOARD being populated for them, in which case the system does use ID_NET_NAME_PATH for the name

Still rabbit-holing on this it seems there may be a Dell BIOS toggle that either enables or disables exposing the ACPI firmware IDs to the OS. Info from this page says the BIOS.IntegratedDevices.NicAcpi setting:

Enable/Disables NIC ACPI device name information.  When set to Enabled, publish ACPI device name for NICs on PCIe slots.  Warning: Changing this setting will cause the NIC name to change under Linux.

Anecdotally the debian version (presumably the systemd version) also plays a role in whether PATH or ONBOARD id's are used, with bookworm using the ONBOARD (acpi id's) if available.

But this toggle may give us some scope to control things, we could disable and force PATH IDs to be picked. Or make sure it's enabled and use the ACPI IDs if we can find a way to get them from Redfish. The curiosity of the "np0" being present or not exists on both and would remain to be solved.

Another complication I see is that on a SuperMicro device there is no ID_NET_NAME_ONBOARD populated, whereas on a Dell there is (though we can perhaps disable with above BIOS toggle). Instead it's using ID_NET_NAME_SLOT which the Dell's don't have. See P71018.