Page MenuHomePhabricator

NIC misassigned (double entries) by jessie installer
Closed, ResolvedPublic

Description

we've seen this recurring with both broadcom and intel cards where udev persistent rules would write the same entry for a NIC twice thus making it appear as ethN with N > 0 and in turn making the install fail. still unknown if it is udev or installer etc

e.g. on restbase1002

# cat /etc/udev/rules.d/70-persistent-net.rules 
# This file was automatically generated by the /lib/udev/write_net_rules
# program, run by the persistent-net-generator.rules rules file.
#
# You can modify it, as long as you keep each rule on a single
# line, and change only the value of the NAME= key.

# PCI device 0x14e4:0x168e (bnx2x)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="8c:dc:d4:00:11:7c", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME="eth1"

# PCI device 0x14e4:0x168e (bnx2x)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="8c:dc:d4:00:11:78", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME="eth0"

# PCI device 0x14e4:0x168e (bnx2x)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="8c:dc:d4:00:11:78", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME="eth2"

# PCI device 0x14e4:0x168e (bnx2x)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="8c:dc:d4:00:11:7c", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME="eth3"
~ #

Event Timeline

fgiunchedi raised the priority of this task from to Needs Triage.
fgiunchedi updated the task description. (Show Details)
fgiunchedi added a project: acl*sre-team.
fgiunchedi added subscribers: fgiunchedi, Joe.

trying to boot up restbase1001 I've seen it timeout on initial dhcp at boot, most of the time it works though as below:

Feb 23 15:39:09 carbon dhcpd: DHCPOFFER on 10.64.0.220 to 8c:dc:d4:00:0c:b0 via 10.64.0.3
Feb 23 15:39:09 carbon dhcpd: DHCPDISCOVER from 8c:dc:d4:00:0c:b0 via 10.64.0.2
Feb 23 15:39:09 carbon dhcpd: DHCPOFFER on 10.64.0.220 to 8c:dc:d4:00:0c:b0 via 10.64.0.2
Feb 23 15:39:13 carbon dhcpd: DHCPREQUEST for 10.64.0.220 (208.80.154.10) from 8c:dc:d4:00:0c:b0 via 10.64.0.3
Feb 23 15:39:13 carbon dhcpd: DHCPACK on 10.64.0.220 to 8c:dc:d4:00:0c:b0 via 10.64.0.3
Feb 23 15:39:13 carbon dhcpd: DHCPREQUEST for 10.64.0.220 (208.80.154.10) from 8c:dc:d4:00:0c:b0 via 10.64.0.2
Feb 23 15:39:13 carbon dhcpd: DHCPACK on 10.64.0.220 to 8c:dc:d4:00:0c:b0 via 10.64.0.2
Feb 23 15:39:19 carbon atftpd[1103]: Serving jessie-installer/debian-installer/amd64/pxelinux.0 to 10.64.0.220:2072
Feb 23 15:39:19 carbon atftpd[1103]: Serving jessie-installer/debian-installer/amd64/pxelinux.0 to 10.64.0.220:2073
Feb 23 15:39:24 carbon atftpd[1103]: timeout: retrying...
Feb 23 15:39:24 carbon atftpd[1103]: timeout: retrying...
Feb 23 15:39:29 carbon atftpd[1103]: timeout: retrying...
Feb 23 15:39:34 carbon atftpd[1103]: timeout: retrying...
Feb 23 15:39:35 carbon atftpd[1103]: Serving jessie-installer/ldlinux.c32 to 10.64.0.220:49152
Feb 23 15:39:37 carbon atftpd[1103]: Serving jessie-installer/pxelinux.cfg/ttyS1-115200 to 10.64.0.220:49153
Feb 23 15:39:37 carbon atftpd[1103]: Serving jessie-installer/pxelinux.cfg/boot.txt to 10.64.0.220:49154
Feb 23 15:39:39 carbon atftpd[1103]: timeout: retrying...
Feb 23 15:39:44 carbon atftpd[1103]: timeout: retrying...
Feb 23 15:39:48 carbon atftpd[1103]: Serving jessie-installer/debian-installer/amd64/linux to 10.64.0.220:49155

also seemingly fails to load firmware for bnx2x

~ # dmesg | grep -e eth -e bnx
[    2.749242] bnx2x: Broadcom NetXtreme II 5771x/578xx 10/20-Gigabit Ethernet Driver bnx2x 1.78.19-0 (2014/02/10)
[    2.798363] bnx2x 0000:03:00.0: msix capability found
[    2.856489] bnx2x 0000:03:00.0: part number 0-0-0-0
[    2.962638] bnx2x 0000:03:00.0: irq 90 for MSI/MSI-X
[    2.962651] bnx2x 0000:03:00.0: irq 91 for MSI/MSI-X
[    2.962673] bnx2x 0000:03:00.0: irq 92 for MSI/MSI-X
[    2.962685] bnx2x 0000:03:00.0: irq 93 for MSI/MSI-X
[    2.962697] bnx2x 0000:03:00.0: irq 94 for MSI/MSI-X
[    2.962708] bnx2x 0000:03:00.0: irq 95 for MSI/MSI-X
[    2.962719] bnx2x 0000:03:00.0: irq 96 for MSI/MSI-X
[    2.962730] bnx2x 0000:03:00.0: irq 97 for MSI/MSI-X
[    2.962741] bnx2x 0000:03:00.0: irq 98 for MSI/MSI-X
[    2.962753] bnx2x 0000:03:00.0: irq 99 for MSI/MSI-X
[    2.963883] bnx2x 0000:03:00.1: msix capability found
[    2.964281] bnx2x 0000:03:00.1: part number 0-0-0-0
[    3.046719] bnx2x 0000:03:00.1: irq 100 for MSI/MSI-X
[    3.046733] bnx2x 0000:03:00.1: irq 101 for MSI/MSI-X
[    3.046745] bnx2x 0000:03:00.1: irq 102 for MSI/MSI-X
[    3.046756] bnx2x 0000:03:00.1: irq 103 for MSI/MSI-X
[    3.046767] bnx2x 0000:03:00.1: irq 104 for MSI/MSI-X
[    3.046779] bnx2x 0000:03:00.1: irq 105 for MSI/MSI-X
[    3.046792] bnx2x 0000:03:00.1: irq 106 for MSI/MSI-X
[    3.046804] bnx2x 0000:03:00.1: irq 107 for MSI/MSI-X
[    3.046816] bnx2x 0000:03:00.1: irq 108 for MSI/MSI-X
[    3.046827] bnx2x 0000:03:00.1: irq 109 for MSI/MSI-X
[    5.733535] systemd-udevd[219]: renamed network interface eth1 to eth2
[    6.734611] systemd-udevd[217]: renamed network interface eth0 to eth3
[    8.043710] bnx2x 0000:03:00.1: firmware: failed to load bnx2x/bnx2x-e2-7.8.19.0.fw (-2)
[    8.043716] bnx2x 0000:03:00.1: Direct firmware load failed with error -2
[    8.043719] bnx2x 0000:03:00.1: Falling back to user helper
[    8.044341] bnx2x: [bnx2x_func_hw_init:5506(eth2)]Error loading firmware
[    8.044354] bnx2x: [bnx2x_nic_load:2685(eth2)]HW init failed, aborting
[    8.207856] bnx2x 0000:03:00.0: firmware: failed to load bnx2x/bnx2x-e2-7.8.19.0.fw (-2)
[    8.207862] bnx2x 0000:03:00.0: Direct firmware load failed with error -2
[    8.207865] bnx2x 0000:03:00.0: Falling back to user helper
[    8.208314] bnx2x: [bnx2x_func_hw_init:5506(eth3)]Error loading firmware
[    8.208325] bnx2x: [bnx2x_nic_load:2685(eth3)]HW init failed, aborting
[   23.109945] bnx2x: Broadcom NetXtreme II 5771x/578xx 10/20-Gigabit Ethernet Driver bnx2x 1.78.19-0 (2014/02/10)
[   23.110111] bnx2x 0000:03:00.0: msix capability found
[   23.123092] bnx2x 0000:03:00.0: part number 0-0-0-0
[   23.202932] bnx2x 0000:03:00.0: irq 90 for MSI/MSI-X
[   23.202948] bnx2x 0000:03:00.0: irq 91 for MSI/MSI-X
[   23.202973] bnx2x 0000:03:00.0: irq 92 for MSI/MSI-X
[   23.202986] bnx2x 0000:03:00.0: irq 93 for MSI/MSI-X
[   23.203000] bnx2x 0000:03:00.0: irq 94 for MSI/MSI-X
[   23.203014] bnx2x 0000:03:00.0: irq 95 for MSI/MSI-X
[   23.203027] bnx2x 0000:03:00.0: irq 96 for MSI/MSI-X
[   23.203040] bnx2x 0000:03:00.0: irq 97 for MSI/MSI-X
[   23.203054] bnx2x 0000:03:00.0: irq 98 for MSI/MSI-X
[   23.203070] bnx2x 0000:03:00.0: irq 99 for MSI/MSI-X
[   23.204396] bnx2x 0000:03:00.1: msix capability found
[   23.219124] bnx2x 0000:03:00.1: part number 0-0-0-0
[   23.247127] systemd-udevd[1533]: renamed network interface eth0 to eth3
[   23.299052] bnx2x 0000:03:00.1: irq 100 for MSI/MSI-X
[   23.299069] bnx2x 0000:03:00.1: irq 101 for MSI/MSI-X
[   23.299083] bnx2x 0000:03:00.1: irq 102 for MSI/MSI-X
[   23.299095] bnx2x 0000:03:00.1: irq 103 for MSI/MSI-X
[   23.299108] bnx2x 0000:03:00.1: irq 104 for MSI/MSI-X
[   23.299121] bnx2x 0000:03:00.1: irq 105 for MSI/MSI-X
[   23.299144] bnx2x 0000:03:00.1: irq 106 for MSI/MSI-X
[   23.299157] bnx2x 0000:03:00.1: irq 107 for MSI/MSI-X
[   23.299171] bnx2x 0000:03:00.1: irq 108 for MSI/MSI-X
[   23.299185] bnx2x 0000:03:00.1: irq 109 for MSI/MSI-X
[   23.343288] systemd-udevd[1533]: renamed network interface eth0 to eth2
[   23.379305] bnx2x 0000:03:00.0: firmware: direct-loading firmware bnx2x/bnx2x-e2-7.8.19.0.fw
[   23.887483] bnx2x 0000:03:00.0 eth3: using MSI-X  IRQs: sp 90  fp[0] 92 ... fp[7] 99
[   24.071774] bnx2x 0000:03:00.0 eth3: NIC Link is Up, 10000 Mbps full duplex, Flow control: ON - receive & transmit
[   26.085841] bnx2x 0000:03:00.1 eth2: using MSI-X  IRQs: sp 100  fp[0] 102 ... fp[7] 109
[   26.220112] IPv6: ADDRCONF(NETDEV_UP): eth2: link is not ready
[   30.157545] bnx2x 0000:03:00.0 eth3: using MSI-X  IRQs: sp 90  fp[0] 92 ... fp[7] 99
[   30.342084] bnx2x 0000:03:00.0 eth3: NIC Link is Up, 10000 Mbps full duplex, Flow control: ON - receive & transmit
[   31.562994] bnx2x 0000:03:00.0 eth3: using MSI-X  IRQs: sp 90  fp[0] 92 ... fp[7] 99
[   31.747527] bnx2x 0000:03:00.0 eth3: NIC Link is Up, 10000 Mbps full duplex, Flow control: ON - receive & transmit
[   43.838969] bnx2x 0000:03:00.1 eth2: using MSI-X  IRQs: sp 100  fp[0] 102 ... fp[7] 109
[   43.969909] IPv6: ADDRCONF(NETDEV_UP): eth2: link is not ready
[   53.048737] bnx2x 0000:03:00.1 eth2: using MSI-X  IRQs: sp 100  fp[0] 102 ... fp[7] 109
[   53.183160] IPv6: ADDRCONF(NETDEV_UP): eth2: link is not ready
[   57.120673] bnx2x 0000:03:00.0 eth3: using MSI-X  IRQs: sp 90  fp[0] 92 ... fp[7] 99
[   57.305214] bnx2x 0000:03:00.0 eth3: NIC Link is Up, 10000 Mbps full duplex, Flow control: ON - receive & transmit
~ #

more from /var/log/syslog, I've tried passing rd.udev.log-priority=debug and udev.log-priority=debug at boot but doesn't seem to have an effect

~ # grep -e bnx -e udev -e eth /var/log/syslog
Feb 23 19:35:45 kernel: [    0.000000] Command line: BOOT_IMAGE=debian-installer/amd64/linux initrd=debian-installer/amd64/initrd.gz vga=normal auto-install/enable=true preseed/url=http://apt.wikimedia.org/autoinstall/preseed.cfg DEBCONF_DEBUG=5 netcfg/choose_interface=auto netcfg/get_hostname=unassigned netcfg/get_domain=unassigned netcfg/dhcp_timeout=60 --- console=ttyS1,115200n8 udev.log-priority=debug rd.udev.log-priority=debug
Feb 23 19:35:45 kernel: [    0.000000] Kernel command line: BOOT_IMAGE=debian-installer/amd64/linux initrd=debian-installer/amd64/initrd.gz vga=normal auto-install/enable=true preseed/url=http://apt.wikimedia.org/autoinstall/preseed.cfg DEBCONF_DEBUG=5 netcfg/choose_interface=auto netcfg/get_hostname=unassigned netcfg/get_domain=unassigned netcfg/dhcp_timeout=60 --- console=ttyS1,115200n8 udev.log-priority=debug rd.udev.log-priority=debug
Feb 23 19:35:45 kernel: [    2.534384] systemd-udevd[207]: starting version 215
Feb 23 19:35:45 kernel: [    2.559432] random: systemd-udevd urandom read with 2 bits of entropy available
Feb 23 19:35:45 kernel: [    2.768438] bnx2x: Broadcom NetXtreme II 5771x/578xx 10/20-Gigabit Ethernet Driver bnx2x 1.78.19-0 (2014/02/10)
Feb 23 19:35:45 kernel: [    2.819320] bnx2x 0000:03:00.0: msix capability found
Feb 23 19:35:45 kernel: [    2.819682] bnx2x 0000:03:00.0: part number 0-0-0-0
Feb 23 19:35:45 kernel: [    2.898221] bnx2x 0000:03:00.0: irq 90 for MSI/MSI-X
Feb 23 19:35:45 kernel: [    2.898228] bnx2x 0000:03:00.0: irq 91 for MSI/MSI-X
Feb 23 19:35:45 kernel: [    2.898235] bnx2x 0000:03:00.0: irq 92 for MSI/MSI-X
Feb 23 19:35:45 kernel: [    2.898242] bnx2x 0000:03:00.0: irq 93 for MSI/MSI-X
Feb 23 19:35:45 kernel: [    2.898249] bnx2x 0000:03:00.0: irq 94 for MSI/MSI-X
Feb 23 19:35:45 kernel: [    2.898255] bnx2x 0000:03:00.0: irq 95 for MSI/MSI-X
Feb 23 19:35:45 kernel: [    2.898262] bnx2x 0000:03:00.0: irq 96 for MSI/MSI-X
Feb 23 19:35:45 kernel: [    2.898268] bnx2x 0000:03:00.0: irq 97 for MSI/MSI-X
Feb 23 19:35:45 kernel: [    2.898275] bnx2x 0000:03:00.0: irq 98 for MSI/MSI-X
Feb 23 19:35:45 kernel: [    2.898281] bnx2x 0000:03:00.0: irq 99 for MSI/MSI-X
Feb 23 19:35:45 kernel: [    2.899004] bnx2x 0000:03:00.1: msix capability found
Feb 23 19:35:45 kernel: [    2.899343] bnx2x 0000:03:00.1: part number 0-0-0-0
Feb 23 19:35:45 kernel: [    2.978287] bnx2x 0000:03:00.1: irq 100 for MSI/MSI-X
Feb 23 19:35:45 kernel: [    2.978295] bnx2x 0000:03:00.1: irq 101 for MSI/MSI-X
Feb 23 19:35:45 kernel: [    2.978301] bnx2x 0000:03:00.1: irq 102 for MSI/MSI-X
Feb 23 19:35:45 kernel: [    2.978308] bnx2x 0000:03:00.1: irq 103 for MSI/MSI-X
Feb 23 19:35:45 kernel: [    2.978317] bnx2x 0000:03:00.1: irq 104 for MSI/MSI-X
Feb 23 19:35:45 kernel: [    2.978323] bnx2x 0000:03:00.1: irq 105 for MSI/MSI-X
Feb 23 19:35:45 kernel: [    2.978331] bnx2x 0000:03:00.1: irq 106 for MSI/MSI-X
Feb 23 19:35:45 kernel: [    2.978338] bnx2x 0000:03:00.1: irq 107 for MSI/MSI-X
Feb 23 19:35:45 kernel: [    2.978345] bnx2x 0000:03:00.1: irq 108 for MSI/MSI-X
Feb 23 19:35:45 kernel: [    2.978351] bnx2x 0000:03:00.1: irq 109 for MSI/MSI-X
Feb 23 19:35:46 debconf: --> METAGET debian-installer/ethdetect/title Description
Feb 23 19:35:46 debconf: --> METAGET debian-installer/ethdetect/title Description
Feb 23 19:35:46 debconf: --> METAGET debian-installer/ethdetect/title Description
Feb 23 19:35:46 main-menu[436]: INFO: Menu item 'ethdetect' selected
Feb 23 19:35:46 debconf: --> SETTITLE debian-installer/ethdetect/title
Feb 23 19:35:46 debconf: --> PROGRESS START 0 1000 ethdetect/detect_progress_title
Feb 23 19:35:46 net/hw-detect.hotplug: Detected hotpluggable network interface eth2
Feb 23 19:35:46 kernel: [    5.737118] systemd-udevd[219]: renamed network interface eth0 to eth2
Feb 23 19:35:47 net/hw-detect.hotplug: Detected hotpluggable network interface eth3
Feb 23 19:35:47 kernel: [    6.730367] systemd-udevd[214]: renamed network interface eth1 to eth3
Feb 23 19:35:47 apt-install: Queueing package udev for later installation
Feb 23 19:35:48 check-missing-firmware: /dev/.udev/firmware-missing does not exist, skipping
Feb 23 19:35:48 check-missing-firmware: /run/udev/firmware-missing does not exist, skipping
Feb 23 19:35:49 debconf: --> GET ethdetect/prompt_missing_firmware
Feb 23 19:35:49 check-missing-firmware: taking network interface eth2 up/down
Feb 23 19:35:49 kernel: [    8.039390] bnx2x 0000:03:00.0: firmware: failed to load bnx2x/bnx2x-e2-7.8.19.0.fw (-2)
Feb 23 19:35:49 kernel: [    8.039396] bnx2x 0000:03:00.0: Direct firmware load failed with error -2
Feb 23 19:35:49 kernel: [    8.039399] bnx2x 0000:03:00.0: Falling back to user helper
Feb 23 19:35:49 kernel: [    8.040044] bnx2x: [bnx2x_func_hw_init:5506(eth2)]Error loading firmware
Feb 23 19:35:49 kernel: [    8.040057] bnx2x: [bnx2x_nic_load:2685(eth2)]HW init failed, aborting
Feb 23 19:35:49 check-missing-firmware: taking network interface eth3 up/down
Feb 23 19:35:49 kernel: [    8.203520] bnx2x 0000:03:00.1: firmware: failed to load bnx2x/bnx2x-e2-7.8.19.0.fw (-2)
Feb 23 19:35:49 kernel: [    8.203526] bnx2x 0000:03:00.1: Direct firmware load failed with error -2
Feb 23 19:35:49 kernel: [    8.203529] bnx2x 0000:03:00.1: Falling back to user helper
Feb 23 19:35:49 kernel: [    8.204036] bnx2x: [bnx2x_func_hw_init:5506(eth3)]Error loading firmware
Feb 23 19:35:49 kernel: [    8.204047] bnx2x: [bnx2x_nic_load:2685(eth3)]HW init failed, aborting
Feb 23 19:35:50 check-missing-firmware: looking for firmware file bnx2x/bnx2x-e2-7.8.19.0.fw requested by bnx2x
Feb 23 19:35:50 check-missing-firmware: looking for firmware file bnx2x/bnx2x-e2-7.8.19.0.fw requested by bnx2x
Feb 23 19:35:50 check-missing-firmware: /dev/.udev/firmware-missing does not exist, skipping
Feb 23 19:35:50 check-missing-firmware: /run/udev/firmware-missing does not exist, skipping
Feb 23 19:35:50 check-missing-firmware: missing firmware files (bnx2x/bnx2x-e2-7.8.19.0.fw bnx2x/bnx2x-e2-7.8.19.0.fw) for bnx2x bnx2x
Feb 23 19:35:51 check-missing-firmware: installing firmware package /firmware/firmware-bnx2x_0.43_all.deb
Feb 23 19:36:04 check-missing-firmware: removing and loading kernel module bnx2x
Feb 23 19:36:04 kernel: [   23.329656] bnx2x: Broadcom NetXtreme II 5771x/578xx 10/20-Gigabit Ethernet Driver bnx2x 1.78.19-0 (2014/02/10)
Feb 23 19:36:04 kernel: [   23.329831] bnx2x 0000:03:00.0: msix capability found
Feb 23 19:36:04 kernel: [   23.342973] bnx2x 0000:03:00.0: part number 0-0-0-0
Feb 23 19:36:04 kernel: [   23.422820] bnx2x 0000:03:00.0: irq 90 for MSI/MSI-X
Feb 23 19:36:04 kernel: [   23.422836] bnx2x 0000:03:00.0: irq 91 for MSI/MSI-X
Feb 23 19:36:04 kernel: [   23.422852] bnx2x 0000:03:00.0: irq 92 for MSI/MSI-X
Feb 23 19:36:04 kernel: [   23.422866] bnx2x 0000:03:00.0: irq 93 for MSI/MSI-X
Feb 23 19:36:04 kernel: [   23.422879] bnx2x 0000:03:00.0: irq 94 for MSI/MSI-X
Feb 23 19:36:04 kernel: [   23.422892] bnx2x 0000:03:00.0: irq 95 for MSI/MSI-X
Feb 23 19:36:04 kernel: [   23.422906] bnx2x 0000:03:00.0: irq 96 for MSI/MSI-X
Feb 23 19:36:04 kernel: [   23.422920] bnx2x 0000:03:00.0: irq 97 for MSI/MSI-X
Feb 23 19:36:04 kernel: [   23.422933] bnx2x 0000:03:00.0: irq 98 for MSI/MSI-X
Feb 23 19:36:04 kernel: [   23.422946] bnx2x 0000:03:00.0: irq 99 for MSI/MSI-X
Feb 23 19:36:04 kernel: [   23.424205] bnx2x 0000:03:00.1: msix capability found
Feb 23 19:36:04 kernel: [   23.439013] bnx2x 0000:03:00.1: part number 0-0-0-0
Feb 23 19:36:04 net/hw-detect.hotplug: Detected hotpluggable network interface eth2
Feb 23 19:36:04 kernel: [   23.499019] systemd-udevd[1544]: renamed network interface eth0 to eth2
Feb 23 19:36:04 check-missing-firmware: taking network interface eth2 up/down
Feb 23 19:36:04 kernel: [   23.519002] bnx2x 0000:03:00.1: irq 100 for MSI/MSI-X
Feb 23 19:36:04 kernel: [   23.519019] bnx2x 0000:03:00.1: irq 101 for MSI/MSI-X
Feb 23 19:36:04 kernel: [   23.519032] bnx2x 0000:03:00.1: irq 102 for MSI/MSI-X
Feb 23 19:36:04 kernel: [   23.519045] bnx2x 0000:03:00.1: irq 103 for MSI/MSI-X
Feb 23 19:36:04 kernel: [   23.519057] bnx2x 0000:03:00.1: irq 104 for MSI/MSI-X
Feb 23 19:36:04 kernel: [   23.519069] bnx2x 0000:03:00.1: irq 105 for MSI/MSI-X
Feb 23 19:36:04 kernel: [   23.519086] bnx2x 0000:03:00.1: irq 106 for MSI/MSI-X
Feb 23 19:36:04 kernel: [   23.519101] bnx2x 0000:03:00.1: irq 107 for MSI/MSI-X
Feb 23 19:36:04 kernel: [   23.519114] bnx2x 0000:03:00.1: irq 108 for MSI/MSI-X
Feb 23 19:36:04 kernel: [   23.519128] bnx2x 0000:03:00.1: irq 109 for MSI/MSI-X
Feb 23 19:36:04 net/hw-detect.hotplug: Detected hotpluggable network interface eth3
Feb 23 19:36:04 kernel: [   23.563167] systemd-udevd[1544]: renamed network interface eth0 to eth3
Feb 23 19:36:04 kernel: [   23.599247] bnx2x 0000:03:00.0: firmware: direct-loading firmware bnx2x/bnx2x-e2-7.8.19.0.fw
Feb 23 19:36:05 kernel: [   24.107340] bnx2x 0000:03:00.0 eth2: using MSI-X  IRQs: sp 90  fp[0] 92 ... fp[7] 99
Feb 23 19:36:05 kernel: [   24.291671] bnx2x 0000:03:00.0 eth2: NIC Link is Up, 10000 Mbps full duplex, Flow control: ON - receive & transmit
Feb 23 19:36:05 check-missing-firmware: taking network interface eth3 up/down
Feb 23 19:36:06 kernel: [   25.220628] bnx2x 0000:03:00.1 eth3: using MSI-X  IRQs: sp 100  fp[0] 102 ... fp[7] 109
Feb 23 19:36:06 kernel: [   25.354909] IPv6: ADDRCONF(NETDEV_UP): eth3: link is not ready
Feb 23 19:36:07 check-missing-firmware: looking for firmware file bnx2x/bnx2x-e2-7.8.19.0.fw requested by bnx2x
Feb 23 19:36:07 check-missing-firmware: looking for firmware file bnx2x/bnx2x-e2-7.8.19.0.fw requested by bnx2x
Feb 23 19:36:07 check-missing-firmware: /dev/.udev/firmware-missing does not exist, skipping
Feb 23 19:36:07 check-missing-firmware: /run/udev/firmware-missing does not exist, skipping
Feb 23 19:36:07 debconf: --> METAGET debian-installer/ethdetect/title Description
Feb 23 19:36:07 debconf: --> METAGET debian-installer/ethdetect/title Description
Feb 23 19:36:07 netcfg[1674]: INFO: Activating interface eth2
Feb 23 19:36:08 kernel: [   27.374434] bnx2x 0000:03:00.0 eth2: using MSI-X  IRQs: sp 90  fp[0] 92 ... fp[7] 99
Feb 23 19:36:08 debconf: --> SUBST netcfg/link_detect_progress interface eth2
Feb 23 19:36:08 debconf: Adding [interface] -> [eth2]
Feb 23 19:36:08 kernel: [   27.558954] bnx2x 0000:03:00.0 eth2: NIC Link is Up, 10000 Mbps full duplex, Flow control: ON - receive & transmit
Feb 23 19:36:08 netcfg[1674]: INFO: ethtool-lite: eth2 is connected.
Feb 23 19:36:08 netcfg[1674]: INFO: Found link on eth2
Feb 23 19:36:08 netcfg[1674]: INFO: found link on interface eth2, making it the default.
Feb 23 19:36:08 netcfg[1674]: INFO: Taking down interface eth2
Feb 23 19:36:09 debconf: --> SET netcfg/choose_interface eth2: Broadcom Corporation NetXtreme II BCM57810 10 Gigabit Ethernet
Feb 23 19:36:09 netcfg[1674]: INFO: Taking down interface eth3
Feb 23 19:36:09 debconf: --> SUBST netcfg/choose_interface ifchoices eth2: Broadcom Corporation NetXtreme II BCM57810 10 Gigabit Ethernet, eth3: Broadcom Corporation NetXtreme II BCM57810 10 Gigabit Ethernet
Feb 23 19:36:09 debconf: Adding [ifchoices] -> [eth2: Broadcom Corporation NetXtreme II BCM57810 10 Gigabit Ethernet, eth3: Broadcom Corporation NetXtreme II BCM57810 10 Gigabit Ethernet]
Feb 23 19:36:09 debconf: <-- 0 eth2: Broadcom Corporation NetXtreme II BCM57810 10 Gigabit Ethernet
Feb 23 19:36:09 netcfg[1674]: INFO: Activating interface eth2
Feb 23 19:36:09 kernel: [   28.779878] bnx2x 0000:03:00.0 eth2: using MSI-X  IRQs: sp 90  fp[0] 92 ... fp[7] 99
Feb 23 19:36:10 netcfg[1674]: DEBUG: Want link on eth2
Feb 23 19:36:10 debconf: --> SUBST netcfg/link_detect_progress interface eth2
Feb 23 19:36:10 debconf: Adding [interface] -> [eth2]
Feb 23 19:36:10 kernel: [   28.964394] bnx2x 0000:03:00.0 eth2: NIC Link is Up, 10000 Mbps full duplex, Flow control: ON - receive & transmit
Feb 23 19:36:10 netcfg[1674]: INFO: ethtool-lite: eth2 is connected.
Feb 23 19:36:10 netcfg[1674]: INFO: Found link on eth2
Feb 23 19:36:10 netcfg[1674]: DEBUG: Commencing network autoconfiguration on eth2
Feb 23 19:36:10 netcfg[1674]: DEBUG: nc_v6_interface_configured(eth2, scope local)
Feb 23 19:36:10 netcfg[1674]: DEBUG: Running ip addr show eth2 to look for address
Feb 23 19:36:10 netcfg[1674]: DEBUG: ip line: 4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq qlen 1000
Feb 23 19:36:10 netcfg[1674]: DEBUG: ip line:     link/ether 8c:dc:d4:00:0c:b0 brd ff:ff:ff:ff:ff:ff
Feb 23 19:36:10 netcfg[1674]: DEBUG: nc_v6_interface_configured(eth2, scope local)
Feb 23 19:36:10 netcfg[1674]: DEBUG: Running ip addr show eth2 to look for address
Feb 23 19:36:10 netcfg[1674]: DEBUG: ip line: 4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq qlen 1000
Feb 23 19:36:10 netcfg[1674]: DEBUG: ip line:     link/ether 8c:dc:d4:00:0c:b0 brd ff:ff:ff:ff:ff:ff
Feb 23 19:36:11 netcfg[1674]: DEBUG: nc_v6_interface_configured(eth2, scope local)
Feb 23 19:36:11 netcfg[1674]: DEBUG: Running ip addr show eth2 to look for address
Feb 23 19:36:11 netcfg[1674]: DEBUG: ip line: 4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq qlen 1000
Feb 23 19:36:11 netcfg[1674]: DEBUG: ip line:     link/ether 8c:dc:d4:00:0c:b0 brd ff:ff:ff:ff:ff:ff
Feb 23 19:36:11 netcfg[1674]: DEBUG: nc_v6_interface_configured(eth2, scope local)
Feb 23 19:36:11 netcfg[1674]: DEBUG: Running ip addr show eth2 to look for address
Feb 23 19:36:11 netcfg[1674]: DEBUG: ip line: 4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq qlen 1000
Feb 23 19:36:11 netcfg[1674]: DEBUG: ip line:     link/ether 8c:dc:d4:00:0c:b0 brd ff:ff:ff:ff:ff:ff
Feb 23 19:36:11 netcfg[1674]: DEBUG: nc_v6_interface_configured(eth2, scope local)
Feb 23 19:36:11 netcfg[1674]: DEBUG: Running ip addr show eth2 to look for address
Feb 23 19:36:11 netcfg[1674]: DEBUG: ip line: 4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq qlen 1000
Feb 23 19:36:11 netcfg[1674]: DEBUG: ip line:     link/ether 8c:dc:d4:00:0c:b0 brd ff:ff:ff:ff:ff:ff
Feb 23 19:36:11 netcfg[1674]: DEBUG: nc_v6_interface_configured(eth2, scope local)
Feb 23 19:36:11 netcfg[1674]: DEBUG: Running ip addr show eth2 to look for address
Feb 23 19:36:11 netcfg[1674]: DEBUG: ip line: 4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq qlen 1000
Feb 23 19:36:11 netcfg[1674]: DEBUG: ip line:     link/ether 8c:dc:d4:00:0c:b0 brd ff:ff:ff:ff:ff:ff
Feb 23 19:36:11 netcfg[1674]: DEBUG: Running rdisc6 -1 -r 1 -w 500 -n eth2 to get IPv6 config flags
Feb 23 19:36:12 netcfg[1674]: DEBUG: rdisc6 line: Soliciting ff02::2 (ff02::2) on eth2...
Feb 23 19:36:12 netcfg[1674]: DEBUG: rdisc6 line: Soliciting ff02::2 (ff02::2) on eth2...
Feb 23 19:36:13 netcfg[1674]: DEBUG: rdisc6 line: Soliciting ff02::2 (ff02::2) on eth2...
Feb 23 19:36:13 netcfg[1674]: DEBUG: rdisc6 line: Soliciting ff02::2 (ff02::2) on eth2...
Feb 23 19:36:13 netcfg[1674]: DEBUG: nc_v6_interface_configured(eth2, scope global)
Feb 23 19:36:13 netcfg[1674]: DEBUG: Running ip addr show eth2 to look for address
Feb 23 19:36:13 netcfg[1674]: DEBUG: ip line: 4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq qlen 1000
Feb 23 19:36:13 netcfg[1674]: DEBUG: ip line:     link/ether 8c:dc:d4:00:0c:b0 brd ff:ff:ff:ff:ff:ff
Feb 23 19:36:14 netcfg[1674]: DEBUG: nc_v6_interface_configured(eth2, scope global)
Feb 23 19:36:14 netcfg[1674]: DEBUG: Running ip addr show eth2 to look for address
Feb 23 19:36:14 netcfg[1674]: DEBUG: ip line: 4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq qlen 1000
Feb 23 19:36:14 netcfg[1674]: DEBUG: ip line:     link/ether 8c:dc:d4:00:0c:b0 brd ff:ff:ff:ff:ff:ff
Feb 23 19:36:14 netcfg[1674]: DEBUG: nc_v6_interface_configured(eth2, scope global)
Feb 23 19:36:14 netcfg[1674]: DEBUG: Running ip addr show eth2 to look for address
Feb 23 19:36:14 netcfg[1674]: DEBUG: ip line: 4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq qlen 1000
Feb 23 19:36:14 netcfg[1674]: DEBUG: ip line:     link/ether 8c:dc:d4:00:0c:b0 brd ff:ff:ff:ff:ff:ff
Feb 23 19:36:14 netcfg[1674]: DEBUG: nc_v6_interface_configured(eth2, scope global)
Feb 23 19:36:14 netcfg[1674]: DEBUG: Running ip addr show eth2 to look for address
Feb 23 19:36:14 netcfg[1674]: DEBUG: ip line: 4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq qlen 1000
Feb 23 19:36:14 netcfg[1674]: DEBUG: ip line:     link/ether 8c:dc:d4:00:0c:b0 brd ff:ff:ff:ff:ff:ff
Feb 23 19:36:15 netcfg[1674]: DEBUG: nc_v6_interface_configured(eth2, scope global)
Feb 23 19:36:15 netcfg[1674]: DEBUG: Running ip addr show eth2 to look for address
Feb 23 19:36:15 netcfg[1674]: DEBUG: ip line: 4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq qlen 1000
Feb 23 19:36:15 netcfg[1674]: DEBUG: ip line:     link/ether 8c:dc:d4:00:0c:b0 brd ff:ff:ff:ff:ff:ff
Feb 23 19:36:15 netcfg[1674]: DEBUG: nc_v6_interface_configured(eth2, scope global)
Feb 23 19:36:15 netcfg[1674]: DEBUG: Running ip addr show eth2 to look for address
Feb 23 19:36:15 netcfg[1674]: DEBUG: ip line: 4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq qlen 1000
Feb 23 19:36:15 netcfg[1674]: DEBUG: ip line:     link/ether 8c:dc:d4:00:0c:b0 brd ff:ff:ff:ff:ff:ff
Feb 23 19:36:15 udhcpc: Got IP 10.64.0.220 (using eth2) and routing through 10.64.0.1
Feb 23 19:36:18 netcfg[1674]: DEBUG: Writing DHCP stanza for eth2
Feb 23 19:36:18 netcfg[1674]: INFO: Detected eth2 as a hotpluggable device
Feb 23 19:36:18 netcfg[1674]: DEBUG: Writing SLAAC stanza for eth2
Feb 23 19:36:18 debconf: --> METAGET debian-installer/ethdetect/title Description
Feb 23 19:36:18 debconf: --> METAGET debian-installer/ethdetect/title Description
Feb 23 19:36:20 debconf: --> SET partman-auto/method raid
Feb 23 19:36:20 debconf: <-- 10 partman-auto/method doesn't exist
Feb 23 19:36:20 debconf: --> REGISTER debian-installer/dummy partman-auto/method
Feb 23 19:36:20 debconf: --> SET partman-auto/method raid
Feb 23 19:36:20 debconf: --> SUBST partman-auto/method ID partman-auto/method
Feb 23 19:36:20 debconf: Adding [ID] -> [partman-auto/method]
Feb 23 19:36:20 debconf: --> FSET partman-auto/method seen true
Feb 23 19:36:20 debconf: --> SET partman-auto/expert_recipe raid1-root :: 30000	1	30000	raid $primary{ } method{ raid } $lvmignore{ } . 1000	2	1000	raid $primary{ } method{ raid } $lvmignore{ } . 500000	3	-1	raid $primary{ } method{ raid } $lvmignore{ } . 500000   4	-1  ext4 $lvmok{ } $defaultignore{ } lv_name{ var } method{ format } format{ } use_filesystem{ } filesystem{ ext4 } mountpoint{ /var } .
Feb 23 19:36:20 debconf: --> SET partman-auto/expert_recipe raid1-root :: 30000	1	30000	raid $primary{ } method{ raid } $lvmignore{ } . 1000	2	1000	raid $primary{ } method{ raid } $lvmignore{ } . 500000	3	-1	raid $primary{ } method{ raid } $lvmignore{ } . 500000   4	-1  ext4 $lvmok{ } $defaultignore{ } lv_name{ var } method{ format } format{ } use_filesystem{ } filesystem{ ext4 } mountpoint{ /var } .
Feb 23 19:36:21 log-output: ip: can't find device 'eth0'
Feb 23 19:36:22 netcfg[5516]: INFO: Activating interface eth2
Feb 23 19:36:22 debconf: --> SUBST netcfg/link_detect_progress interface eth2
Feb 23 19:36:22 debconf: Adding [interface] -> [eth2]
Feb 23 19:36:22 netcfg[5516]: INFO: ethtool-lite: eth2 is connected.
Feb 23 19:36:22 netcfg[5516]: INFO: Found link on eth2
Feb 23 19:36:22 netcfg[5516]: INFO: found link on interface eth2, making it the default.
Feb 23 19:36:22 netcfg[5516]: INFO: Taking down interface eth2
Feb 23 19:36:22 debconf: --> SET netcfg/choose_interface eth2: Broadcom Corporation NetXtreme II BCM57810 10 Gigabit Ethernet
Feb 23 19:36:22 netcfg[5516]: INFO: Taking down interface eth3
Feb 23 19:36:22 debconf: --> SUBST netcfg/choose_interface ifchoices eth2: Broadcom Corporation NetXtreme II BCM57810 10 Gigabit Ethernet, eth3: Broadcom Corporation NetXtreme II BCM57810 10 Gigabit Ethernet
Feb 23 19:36:22 debconf: Adding [ifchoices] -> [eth2: Broadcom Corporation NetXtreme II BCM57810 10 Gigabit Ethernet, eth3: Broadcom Corporation NetXtreme II BCM57810 10 Gigabit Ethernet]
Feb 23 19:36:22 debconf: <-- 0 eth2: Broadcom Corporation NetXtreme II BCM57810 10 Gigabit Ethernet
Feb 23 19:36:31 netcfg[5516]: INFO: Activating interface eth2
Feb 23 19:36:31 kernel: [   50.774179] bnx2x 0000:03:00.0 eth2: using MSI-X  IRQs: sp 90  fp[0] 92 ... fp[7] 99
Feb 23 19:36:32 debconf: --> SUBST netcfg/link_detect_progress interface eth2
Feb 23 19:36:32 debconf: Adding [interface] -> [eth2]
Feb 23 19:36:32 kernel: [   50.958438] bnx2x 0000:03:00.0 eth2: NIC Link is Up, 10000 Mbps full duplex, Flow control: ON - receive & transmit
Feb 23 19:36:32 netcfg[5516]: INFO: ethtool-lite: eth2 is connected.
Feb 23 19:36:32 netcfg[5516]: INFO: Found link on eth2
Feb 23 19:36:32 netcfg[5516]: INFO: found link on interface eth2, making it the default.
Feb 23 19:36:32 debconf: <-- 0 eth2: Broadcom Corporation NetXtreme II BCM57810 10 Gigabit Ethernet
Feb 23 19:36:32 netcfg[5516]: INFO: Taking down interface eth2
Feb 23 19:36:32 debconf: --> SET netcfg/choose_interface eth2: Broadcom Corporation NetXtreme II BCM57810 10 Gigabit Ethernet
Feb 23 19:36:32 netcfg[5516]: INFO: Taking down interface eth3
Feb 23 19:36:32 debconf: --> SUBST netcfg/choose_interface ifchoices eth2: Broadcom Corporation NetXtreme II BCM57810 10 Gigabit Ethernet, eth3: Broadcom Corporation NetXtreme II BCM57810 10 Gigabit Ethernet
Feb 23 19:36:32 debconf: Adding [ifchoices] -> [eth2: Broadcom Corporation NetXtreme II BCM57810 10 Gigabit Ethernet, eth3: Broadcom Corporation NetXtreme II BCM57810 10 Gigabit Ethernet]
Feb 23 19:36:32 debconf: --> SET netcfg/choose_interface eth2: Broadcom Corporation NetXtreme II BCM57810 10 Gigabit Ethernet
Feb 23 19:36:37 debconf: --> METAGET debian-installer/ethdetect/title Description
Feb 23 19:36:37 debconf: --> METAGET debian-installer/ethdetect/title Description
Feb 23 19:36:40 debconf: --> METAGET debian-installer/ethdetect/title Description
~ #

of course booting with debug makes the race go away, I've been successfully booting restbase1001 5 out of 5 times

to clarify, restbase1002 also had problems but eventually won a race and the installation went through. I've left restbase1001 behind on purpose to be able to debug this, other restbase machines might have been luck too

so the debug hack seems to work reliably in avoiding the race, however it needs to be removed from /etc/default/grub and run update-grub post-installation because d-i carries it over

I can confirm that the debug hack works, sigh.

coren triaged this task as Low priority.Mar 2 2015, 3:50 PM
coren subscribed.

Since there is a workaround, and the bug is filed upstream, I'm setting this to Low priority despite the blocked task being High (nothing we can do directly at this time beyond work around the problem)

@fgiunchedi, do you want to take this want and track upstream if you've subscribed to the bug?

So I debugged this extensively — the Debian bug has all the details of my analysis & a workaround. I've also increased its severity to a Release Critical bug report, with one of the maintainer's agreement, which means there's good chances this will be fixed in time for the jessie release..

I have placed the workaround I made under my (still unpuppetized) update-netboot.sh script and replaced our current jessie d-i snapshot with one that includes the fix. This should be working from now on on (our) new installs.