Page MenuHomePhabricator

d-i fails to install on servers with BRCM 2P 1G BT + 2P 10G SFP NDC
Closed, ResolvedPublic

Description

We have order some new es servers with BRCM 2P 1G BT + 2P 10G SFP NDC Nic's and when doing the installation both on Stretch and Buster it fails with the error below.

───┤ [!!] Configure the network ├─────────────────────┐                                                                           
   │                   Network autoconfiguration failed                    │     
   │ Your network is probably not using the DHCP protocol. Alternatively,  │ 
   │ the DHCP server may be slow or some network hardware is not working   │    
   │ properly.                                                             │ 
   │                                                                       │    
   │                              <Continue>

Event Timeline

Papaul created this task.Jan 10 2020, 11:44 PM
Papaul updated the task description. (Show Details)Jan 10 2020, 11:49 PM
Papaul updated the task description. (Show Details)
Peachey88 renamed this task from Missing Netowrk drivers from Stretch and Buster installer for BRCM 2P 1G BT + 2P 10G SFP NDC to Missing Network drivers from Stretch and Buster installer for BRCM 2P 1G BT + 2P 10G SFP NDC.Jan 11 2020, 4:04 AM

What if you try to configure the network manually rather than using DHCP, does it fail too?

If you send me the MAC addresses, I can try to look for them on the DHCP servers to see if I can at least see the request or if that even doesn't arrive.

@Marostegui the request is not getting to the DHCP server

And setting the IP, GW etc manually doesn't work either?

I didn't try that but you can try with es2024 or es2021.

When reseting es2021 via IDRAC I saw this and after a couple of powercycles the host booted up:

!!!! X64 Exception Type - 0E(#PF - Page-Fault)  CPU Apic ID - 00000000 !!!!
ExceptionData - 0000000000000000  I:0 R:0 U:0 W:0 P:0 PK:0 S:0
RIP  - 0000000048167B33, CS  - 0000000000000038, RFLAGS - 0000000000010012
RAX  - 0000000000000000, RCX - 000000005DA4EFD0, RDX - 000000005D91A028
RBX  - 00000000481482C0, RSP - 0000000048147880, RBP - 000000002BBFF8CC
RSI  - 0000000000000000, RDI - 000000004B6468D8
R8   - 0000000000000013, R9  - 0000000000000000, R10 - 00000000000000C0
R11  - 000000005D91A020, R12 - 0000000000000000, R13 - 0000000000000000
R14  - 0000000000000000, R15 - 0000000000000000
DS   - 0000000000000018, ES  - 0000000000000018, FS  - 0000000000000018
GS   - 0000000000000018, SS  - 0000000000000018
CR0  - 0000000080010013, CR2 - FFFFFFFFE8FD1414, CR3 - 0000000047E01000
CR4  - 0000000000000628, CR8 - 0000000000000000
DR0  - 0000000000000000, DR1 - 0000000000000000, DR2 - 0000000000000000
DR3  - 0000000000000000, DR6 - 00000000FFFF0FF0, DR7 - 0000000000000400
GDTR - 000000004B647D78 0000000000000047, LDTR - 0000000000000000
IDTR - 00000000475FD008 00000000000001FF,   TR - 0000000000000000
FXSAVE_STATE - 00000000481474E0
!!!! Find image based on IP(0x48167B33) u:\Build\DellAtlasPkg\DEBUG_MYTOOLS\X64\DellPkgs\DellCorePkgs\DellRestrictedModulePkg\Override\MdeModulePkg\Core\Dxe\DxeMain\DEBUG\DxeCore.pdb (ImageBase=0000000048148000, EntryPoint=00000000481482C0) !!!!

Once booting it logged this:

0 Non-RAID Disk(s) found on the host adapter
0rNon-RAIDeDisk(s) handledtbyrBIOSBoot Agent
PowerEdge Expandable0RAIDrControlleriBIOSion
1AVirtualsDrive(s)AfoundRonHthePhostpadapter.
1WVirtualeDrive(s).handledfbyuBIOSonMUtility
PowerEdge R740xd
BIOS Version: 2.3.10
Console Redirection Enabled Requested by iDRAC

Attempting PXE Boot
iDRAC IPV4:  10.193.1.72

Initializing Firmware Interfaces...

PowerEdge R740xd - BIOS 2.3.10
A system restart is required. The system detected an exception during the UEFI
pre-boot environment.
-------------------------------------------------------------------------------
Type: Machine Check Error (18) Source: Hardware on BSP
AX=00000000581F873C BX=0000000000000000 SI=0000000000000000 DI=0000000000000000
CX=00000000460843BF DX=0000000045EFE284 R8=00000000001C05B6 R9=00000000001C3B9B
10=00000000000000C0 11=0000000048146160 12=0000000000000000 13=0000000046233AA8
14=00000000588F40D8 15=0000000046233AA0 BP=0000000048146CA8 SP=00000000481461A0
IP=00000000460F4BC0 Flags=00010206  CurrentTPL = 04, LastEventTime FFFFFFFFE812900A
LastMsg:

LBRfr22 460F4BC5 Unknown(ttwgg)
LBRto22 460F4B88 Unknown(ttwgg)
LBRfr21 460F4BC5 Unknown(ttwgg)
LBRto21 460F4B88 Unknown(ttwgg)
LBRfr20 460F4BC5 Unknown(ttwgg)
LBRto20 460F4B88 Unknown(ttwgg)
LBRfr19 460F4BC5 Unknown(ttwgg)
LBRto19 460F4B88 Unknown(ttwgg)
LBRfr18 460F4BC5 Unknown(ttwgg)
LBRto18 460F4B88 Unknown(ttwgg)
LBRfr17 460F4BC5 Unknown(ttwgg)
LBRto17 460F4B88 Unknown(ttwgg)
LBRfr16 460F4BC5 Unknown(ttwgg)
LBRto16 460F4B88 Unknown(ttwgg)
LBRfr15 460F4BC5 Unknown(ttwgg)
LBRto15 460F4B88 Unknown(ttwgg)
LBRfr14 460F4BC5 Unknown(ttwgg)
LBRto14 460F4B88 Unknown(ttwgg)
LBRfr13 460F4BC5 Unknown(ttwgg)
LBRto13 460F4B88 Unknown(ttwgg)
LBRfr12 460F4BC5 Unknown(ttwgg)
LBRto12 460F4B88 Unknown(ttwgg)
LBRfr11 460F4BC5 Unknown(ttwgg)
LBRto11 460F4B88 Unknown(ttwgg)
LBRfr10 460F4BC5 Unknown(ttwgg)
LBRto10 460F4B88 Unknown(ttwgg)
LBRfr9 460F4BC5 Unknown(ttwgg)
LBRto9 460F4B88 Unknown(ttwgg)
LBRfr8 460F4BC5 Unknown(ttwgg)
LBRto8 460F4B88 Unknown(ttwgg)
LBRfr7 460F4BC5 Unknown(ttwgg)
LBRto7 460F4B88 Unknown(ttwgg)
LBRfr6 460F4BC5 Unknown(ttwgg)
LBRto6 460F4B88 Unknown(ttwgg)
LBRfr5 460F4BC5 Unknown(ttwgg)
LBRto5 460F4B88 Unknown(ttwgg)
LBRfr4 460F4BC5 Unknown(ttwgg)
LBRto4 460F4B88 Unknown(ttwgg)
LBRfr3 460F4BC5 Unknown(ttwgg)
LBRto3 460F4B88 Unknown(ttwgg)
LBRfr2 460F4BC5 Unknown(ttwgg)
LBRto2 460F4B88 Unknown(ttwgg)
LBRfr1 460F4BC5 Unknown(ttwgg)
LBRto1 460F4B88 Unknown(ttwgg)
LBRfr0 460F4BC5 Unknown(ttwgg)
LBRto0 460F4B88 Unknown(ttwgg)
-->RIP 460F4BC0 Unknown(ttwgg)
   s00 460F4872 Unknown(ttwgg)
   s01 460F7E1C Unknown(ttwgg)
   s02 460FC36E Unknown(ttwgg)
   s03 460FA8A8 Unknown(ttwgg)
   s04 462EA25D Unknown(ngkkh) +01025D Avago EFI SAS Driver
   s05 462EA3B0 Unknown(ngkkh) +0103B0 Avago EFI SAS Driver
   s06 481698AB DxeCore.efi +0218AB
   s07 48168BE7 DxeCore.efi +020BE7
   s08 481606A2 DxeCore.efi +0186A2
   s09 48168F97 DxeCore.efi +020F97
   s10 481606A2 DxeCore.efi +0186A2
   s11 46DBB033 DellBdsDxe.efi +01A033
   s12 46DBB0A3 DellBdsDxe.efi +01A0A3
   s13 46DBB19D DellBdsDxe.efi +01A19D
   s14 46DA447E DellBdsDxe.efi +00347E
   s15 46DA589A DellBdsDxe.efi +00489A
   s16 48149F5B DxeCore.efi +001F5B
   s17 48148873 DxeCore.efi +000873
   s18 481482DF DxeCore.efi +0002DF
   s19 4B646004 Unknown(hfdzw) +006004

Stack Dump:
  481461A0  00000000581F873C <..X....
  481461A8  00000000460843BF .C.F....
  481461B0  0000000000000958 X.......
  481461B8  00000000460F4872 rH.F....
  481461C0  0000000058072601 .&.X....
  481461C8  0000000045EFE284 ...E....
  481461D0  000000000018613B ;a......
  481461D8  000000004610503B ;P.F....
  481461E0  0000000000000004 ........
  481461E8  00000000460F7E1C .~.F....
  481461F0  0000000058072601 .&.X....
  481461F8  0000000045EFE284 ...E....
  48146200  00000000001C05B6 ........
  48146208  0000000000000000 ........
  48146210  000000005806F020  ..X....
  48146218  00000000001C05B6 ........
  48146220  0000000048146280 .b.H....
  48146228  0000000000000000 ........
  48146230  0000000045EFE280 ...E....
  48146238  FFFFFFFFFFFFFFFF ........
  48146240  0000000058072601 .&.X....
  48146248  0000000000000000 ........
  48146250  0000000000000000 ........
  48146258  00000000460FC36E n..F....
  48146260  00000000460C1C90 ...F....
  48146268  00000000588FEC20  ..X....
  48146270  00000000460DE6A0 ...F....
  48146278  0000000045EFE280 ...E....
  48146280  0000000000000000 ........
  48146288  0000000000000958 X.......
  48146290  0000000000000010 ........
  48146298  000000004816E14D M..H....
  481462A0  0000000000000000 ........
  481462A8  0000000000000000 ........
  481462B0  0100000100000102 ........
  481462B8  C128710000000100 .....q(.
  481462C0  C128737CC02DA078 x.-.|s(.
  481462C8  0000000048154360 `C.H....
  481462D0  1F421028005D1000 ..].(.B.
  481462D8  0000000030430000 ..C0....
  481462E0  0000000000000000 ........
  481462E8  0000000000000000 ........
  481462F0  0000000000030302 ........
  481462F8  0000000000000000 ........
  48146300  0000000000000000 ........
  48146308  0000000000000000 ........
  48146310  0000000000000000 ........
  48146318  0000000000000000 ........
  48146320  0000000000000000 ........
  48146328  0000000000000000 ........
  48146330  0000000000000000 ........
  48146338  0800000000000020  .......
  48146340  500056B374B516FF ...t.V.P
  48146348  0000000000000000 ........
  48146350  0000000000000000 ........
  48146358  0000000000000000 ........
  48146360  0000000000000000 ........
  48146368  0000000000000000 ........
  48146370  0000000000000000 ........
  48146378  0000000000000000 ........
  48146380  0000000838303133 3108....
  48146388  00000000534F4942 BIOS....
  48146390  2E31302E33332E36 6.33.01.
  48146398  302E36312E345F30 0_4.16.0
  481463A0  3078305F30302E37 7.00_0x0
  481463A8  0034303330323136 6120304.
  481463B0  30322F32322F3230 02/22/20
  481463B8  0000000000003931 19......
  481463C0  30322F32322F3230 02/22/20
  481463C8  0000000000003931 19......
  481463D0  00000000524C5443 CTLR....
  481463D8  3037302D38312E35 5.18-070
  481463E0  0000000000000032 2.......
  481463E8  0000000000000000 ........
  481463F0  0000000000000000 ........
  481463F8  3220393220677541 Aug 29 2
  48146400  0000000000383130 018.....
  48146408  37333A35343A3631 16:45:37
  48146410  0000000000000000 ........
  48146418  00000000444C5043 CPLD....
  48146420  0000000030323030 0020....
  48146428  0000000000000000 ........
  48146430  0000000000000000 ........
  48146438  0000000000000000 ........
  48146440  37312F36322F3031 10/26/17
  48146448  0000000000000000 ........
  48146450  33333A30353A3731 17:50:33
  48146458  0000000000000000 ........
  48146460  0000000020505041 APP ....
  48146468  30302E3030332E34 4.300.00
  48146470  000000323533382D -8352...
  48146478  0000000000000000 ........
  48146480  0000000000000000 ........
  48146488  32203230206C754A Jul 02 2
  48146490  0000000000393130 019.....
  48146498  38313A30353A3531 15:50:18
  481464A0  0000000000000000 ........
  481464A8  000000005444564E NVDT....
  481464B0  302E313135312E33 3.1511.0
  481464B8  0000383230302D30 0-0028..
  481464C0  0000000000000000 ........
  481464C8  0000000000000000 ........
  481464D0  322031332074634F Oct 31 2
  481464D8  0000000000373130 017.....
  481464E0  30303A30313A3731 17:10:00
  481464E8  0000000000000000 ........
  481464F0  00000000554D4244 DBMU....
  481464F8  0000000030363030 0060....
  48146500  0000000000000000 ........
  48146508  0000000000000000 ........
  48146510  0000000000000000 ........
  48146518  37312F31332F3830 08/31/17
  48146520  0000000000000000 ........
  48146528  30343A33303A3331 13:03:40
  48146530  0000000000000000 ........
  48146538  0000000057464753 SGFW....
  48146540  0032302E30302E31 1.00.02.
  48146548  0000000000000000 ........
  48146550  0000000000000000 ........
  48146558  0000000000000000 ........
  48146560  3220343220677541 Aug 24 2
  48146568  0000000000333130 013.....
  48146570  38313A36303A3530 05:06:18
  48146578  0000000000000000 ........
  48146580  000000004C425442 BTBL....
  48146588  2E30302E37302E33 3.07.00.
  48146590  00333030302D3030 00-0003.
  48146598  0000000000000000 ........
  481465A0  0000000000000000 ........
  481465A8  32203133206C754A Jul 31 2
  481465B0  0000000000353130 015.....
  481465B8  38313A37343A3431 14:47:18
  481465C0  0000000000000000 ........
  481465C8  0000000000000000 ........
  481465D0  0000000000000000 ........
  481465D8  0000000000000000 ........
  481465E0  0000000000000000 ........
  481465E8  0000000000000000 ........
  481465F0  0000000000000000 ........
  481465F8  0000000000000000 ........
  48146600  0000000000000000 ........
  48146608  0000000000000000 ........
  48146610  0000000000000000 ........
  48146618  0000000000000000 ........
  48146620  0000000000000000 ........
  48146628  0000000000000000 ........
  48146630  0000000000000000 ........
  48146638  0000000000000000 ........
  48146640  0000000000000000 ........
  48146648  0000000000000000 ........
  48146650  0000000000000000 ........
  48146658  0000000000000000 ........
  48146660  0000000000000000 ........
  48146668  0000000000000000 ........
  48146670  0000000000000000 ........
  48146678  0000000000000000 ........
  48146680  0000000000000000 ........
  48146688  0000000000000000 ........
  48146690  0000000000000000 ........
  48146698  0000000000000000 ........
  481466A0  0000000000000000 ........
  481466A8  0000000000000000 ........
  481466B0  0000000000000000 ........
  481466B8  0000000000000000 ........
  481466C0  0000000000000000 ........
  481466C8  0000000000000000 ........
  481466D0  0000000000000000 ........
  481466D8  0000000000000000 ........
  481466E0  0000000000000000 ........
  481466E8  0000000000000000 ........
  481466F0  0000000000000000 ........
  481466F8  0000000000000000 ........
  48146700  0000000000000000 ........
  48146708  0000000000000000 ........
  48146710  0000000000000000 ........
  48146718  0000000000000000 ........
  48146720  0000000000000000 ........
  48146728  0000000000000000 ........
  48146730  0000000000000000 ........
  48146738  0000000000000000 ........
  48146740  0000000000000000 ........
  48146748  0000000000000000 ........
  48146750  0000000000000000 ........
  48146758  0000000000000000 ........
  48146760  0000000000000000 ........
  48146768  0000000000000000 ........
  48146770  0000000000000000 ........
  48146778  0000000000000000 ........
  48146780  0000000000000000 ........
  48146788  0000000000000000 ........
  48146790  0000000000000000 ........
  48146798  0000000000000000 ........
  481467A0  0000000000000000 ........
  481467A8  0000000000000000 ........
  481467B0  0000000000000000 ........
  481467B8  0000000000000000 ........
  481467C0  0000000000000000 ........
  481467C8  0000000000000000 ........
  481467D0  0000000000000000 ........
  481467D8  0000000000000000 ........
  481467E0  0000000000000000 ........
  481467E8  0000000000000000 ........
  481467F0  0000000000000000 ........
  481467F8  0000000000000000 ........
  48146800  0000000000000000 ........
  48146808  4080082000000000 .... ..@
  48146810  3337482043524550 PERC H73
  48146818  7470616441205030 0P Adapt
  48146820  0000000000007265 er......
  48146828  0000000000000000 ........
  48146830  0000000000000000 ........
  48146838  0000000000000000 ........
  48146840  0000000000000000 ........
  48146848  0000000000000000 ........
  48146850  0000000000000000 ........
  48146858  0000000000000000 ........
  48146860  00504F3430314239 9B104OP.
  48146868  0000000000000000 ........
  48146870  0000000000000000 ........
  48146878  0000000000000000 ........
  48146880  25ABDB2500000C3D =...%..%
  48146888  00002000003C03A0 ..<.. ..
  48146890  000D000000000001 ........
  48146898  002000000000000C ...... .
  481468A0  0000000000100800 ........
  481468A8  00000EF700800000 ........
  481468B0  00005E3F1DF45FBF ._..?^..
  481468B8  00042D0700000B07 .....-..
  481468C0  0000000000000007 ........
  481468C8  0000000000030000 ........
  481468D0  00320010012C0006 ..,...2.
  481468D8  0C04041E1E1E1E1E ........
  481468E0  00F0FF0000000100 ........
  481468E8  0000000000000000 ........
  481468F0  001E00000402704C Lp......
  481468F8  00000806000000FE ........
  48146900  000000000E040000 ........
  48146908  00000000000A012C ,.......
  48146910  302E362E352E3532 25.5.6.0
  48146918  0000000000393030 009.....
  48146920  0000000000000000 ........
  48146928  0000000000000000 ........
  48146930  0000000000000000 ........
  48146938  0000000000000000 ........
  48146940  0000000000000000 ........
  48146948  0000000000000000 ........
  48146950  0000000000000000 ........
  48146958  0000000000000000 ........
  48146960  0000000000000000 ........
  48146968  0000000000000000 ........
  48146970  0000000000000000 ........
  48146978  0000000000000000 ........
  48146980  0000000000000000 ........
  48146988  0000000000000000 ........
  48146990  0000000000000000 ........
  48146998  0000000000000000 ........
  481469A0  0000000000000000 ........
  481469A8  0000000000000000 ........
  481469B0  0000000000000000 ........
  481469B8  0000000000000000 ........
  481469C0  0000000000000000 ........
  481469C8  0000000000000000 ........
  481469D0  0000000000000000 ........
  481469D8  0000000000000000 ........
  481469E0  0000000000000000 ........
  481469E8  0000000000000000 ........
  481469F0  0000000000000000 ........
  481469F8  0000000000000000 ........
  48146A00  0000000000000000 ........
  48146A08  0000000000000000 ........
  48146A10  0000000000000000 ........
  48146A18  0000000000000000 ........
  48146A20  0000000000000000 ........
  48146A28  0000000000000000 ........
  48146A30  0203020300220201 ..".....
  48146A38  1006100601040203 ........
  48146A40  0000000010080204 ........
  48146A48  0000000000000000 ........
  48146A50  0200010000FE0100 ........
  48146A58  0020000000001010 ...... .
  48146A60  0000000000000000 ........
  48146A68  0000000000000000 ........
  48146A70  40176AA4078B0000 .....j.@
  48146A78  0000000000000000 ........
  48146A80  0000000000000000 ........
  48146A88  0000000000000000 ........
  48146A90  0000000000000000 ........
  48146A98  000000F8003C3C00 .<<.....
  48146AA0  0000000000000000 ........
  48146AA8  0000000000000000 ........
  48146AB0  0000000000000000 ........
  48146AB8  0000000042520825 %.RB....
  48146AC0  0000000000000000 ........
  48146AC8  0000000000000000 ........
  48146AD0  0000000000000000 ........
  48146AD8  0000000000000000 ........
  48146AE0  0000000000000000 ........
  48146AE8  0000000000000000 ........
  48146AF0  0000000000000000 ........
  48146AF8  0000000000000000 ........
  48146B00  0000000000000000 ........
  48146B08  0000000000000000 ........
  48146B10  0000000000000000 ........
  48146B18  0000000000000000 ........
  48146B20  0000000000000000 ........
  48146B28  0000000000000000 ........
  48146B30  0000000000000000 ........
  48146B38  0000000000000000 ........
  48146B40  0000000000000000 ........
  48146B48  0000000000000000 ........
  48146B50  0000000000000000 ........
  48146B58  0000000000000000 ........
  48146B60  0000000000000000 ........
  48146B68  0000000000000000 ........
  48146B70  0000000000000000 ........
  48146B78  0000000000000000 ........
  48146B80  0000000000000000 ........
  48146B88  0000000000000000 ........
  48146B90  0000000000000000 ........
  48146B98  0000000000000000 ........
  48146BA0  0000000000000000 ........
  48146BA8  0000000000000000 ........
  48146BB0  0000000000000000 ........
  48146BB8  0000000000000000 ........
  48146BC0  0000000000000000 ........
  48146BC8  0000000000000000 ........
  48146BD0  0000000000000000 ........
  48146BD8  0000000000000000 ........
  48146BE0  0000000000000000 ........
  48146BE8  0000000000000000 ........
  48146BF0  0000000000000000 ........
  48146BF8  0000000000000000 ........
  48146C00  0000000000000000 ........
  48146C08  0000000000000000 ........
  48146C10  0000000000000000 ........
  48146C18  0000000000000000 ........
  48146C20  54CD98F0AAE52300 .#.....T
  48146C28  0000020104071E00 ........
  48146C30  0000010000010100 ........
  48146C38  0000000000000001 ........
  48146C40  0000000000000000 ........
  48146C48  00000000582BA520  .+X....
  48146C50  0000000000000000 ........
  48146C58  00000000582BA700 ..+X....
  48146C60  00000000582B3020  0+X....
  48146C68  00000000460FA8A8 ...F....
  48146C70  00000000582BA520  .+X....
  48146C78  00000000460E3A68 h:.F....
  48146C80  8000000000000002 ........
  48146C88  0000000000000000 ........
  48146C90  0000000000000000 ........
  48146C98  00000000462EA25D ]..F....
  48146CA0  00000000588FEC02 ...X....
  48146CA8  00000000588FEC20  ..X....
  48146CB0  0000000048147728 (w.H....
  48146CB8  00000000588FF220  ..X....
  48146CC0  00000000582B6720  g+X....
  48146CC8  0000000000000008 ........
  48146CD0  00000000582B3020  0+X....
  48146CD8  0000000000000000 ........
  48146CE0  0000000000000000 ........
  48146CE8  0000000000000000 ........
  48146CF0  00000000582BA700 ..+X....
  48146CF8  000000004BAD37FE .7.K....
  48146D00  3FFFFFFF00000002 .......?
  48146D08  0000000000000001 ........
  48146D10  0000000001020201 ........
  48146D18  0000000000000000 ........
  48146D20  54CD98F0AAE52300 .#.....T
  48146D28  0000020104071E00 ........
  48146D30  0000010000010100 ........
  48146D38  0100000004000001 ........
  48146D40  0001010000000001 ........
  48146D48  0000000000010001 ........
  48146D50  7F00000000010101 ........
  48146D58  015001000000001E ......P.
  48146D60  0000013C00040075 u...<...
  48146D68  0000000000000000 ........
  48146D70  1F421028005D1000 ..].(.B.
  48146D78  0000000030430000 ..C0....
  48146D80  0000000000000000 ........
  48146D88  0000000000000000 ........
  48146D90  0000000000030302 ........
  48146D98  0000000000000000 ........
  48146DA0  0000000000000000 ........
  48146DA8  0000000000000000 ........
  48146DB0  0000000000000000 ........
  48146DB8  0000000000000000 ........
  48146DC0  0000000000000000 ........
  48146DC8  0000000000000000 ........
  48146DD0  0000000000000000 ........
  48146DD8  0800000000000020  .......
  48146DE0  500056B374B516FF ...t.V.P
  48146DE8  0000000000000000 ........
  48146DF0  0000000000000000 ........
  48146DF8  0000000000000000 ........
  48146E00  0000000000000000 ........
  48146E08  0000000000000000 ........
  48146E10  0000000000000000 ........
  48146E18  0000000000000000 ........
  48146E20  0000000838303133 3108....
  48146E28  00000000534F4942 BIOS....
  48146E30  2E31302E33332E36 6.33.01.
  48146E38  302E36312E345F30 0_4.16.0
  48146E40  3078305F30302E37 7.00_0x0
  48146E48  0034303330323136 6120304.
  48146E50  30322F32322F3230 02/22/20
  48146E58  0000000000003931 19......
  48146E60  30322F32322F3230 02/22/20
  48146E68  0000000000003931 19......
  48146E70  00000000524C5443 CTLR....
  48146E78  3037302D38312E35 5.18-070
  48146E80  0000000000000032 2.......
  48146E88  0000000000000000 ........
  48146E90  0000000000000000 ........
  48146E98  3220393220677541 Aug 29 2
  48146EA0  0000000000383130 018.....
  48146EA8  37333A35343A3631 16:45:37
  48146EB0  0000000000000000 ........
  48146EB8  00000000444C5043 CPLD....
  48146EC0  0000000030323030 0020....
  48146EC8  0000000000000000 ........
  48146ED0  0000000000000000 ........
  48146ED8  0000000000000000 ........
  48146EE0  37312F36322F3031 10/26/17
  48146EE8  0000000000000000 ........
  48146EF0  33333A30353A3731 17:50:33
  48146EF8  0000000000000000 ........
  48146F00  0000000020505041 APP ....
  48146F08  30302E3030332E34 4.300.00
  48146F10  000000323533382D -8352...
  48146F18  0000000000000000 ........
  48146F20  0000000000000000 ........
  48146F28  32203230206C754A Jul 02 2
  48146F30  0000000000393130 019.....
  48146F38  38313A30353A3531 15:50:18
  48146F40  0000000000000000 ........
  48146F48  000000005444564E NVDT....
  48146F50  302E313135312E33 3.1511.0
  48146F58  0000383230302D30 0-0028..
  48146F60  0000000000000000 ........
  48146F68  0000000000000000 ........
  48146F70  322031332074634F Oct 31 2
  48146F78  0000000000373130 017.....
  48146F80  30303A30313A3731 17:10:00
  48146F88  0000000000000000 ........
  48146F90  00000000554D4244 DBMU....
  48146F98  0000000030363030 0060....
  48146FA0  0000000000000000 ........
  48146FA8  0000000000000000 ........
  48146FB0  0000000000000000 ........
  48146FB8  37312F31332F3830 08/31/17
  48146FC0  0000000000000000 ........
  48146FC8  30343A33303A3331 13:03:40
  48146FD0  0000000000000000 ........
  48146FD8  0000000057464753 SGFW....
  48146FE0  0032302E30302E31 1.00.02.
  48146FE8  0000000000000000 ........
  48146FF0  0000000000000000 ........
  48146FF8  0000000000000000 ........
  48147000  3220343220677541 Aug 24 2
  48147008  0000000000333130 013.....
  48147010  38313A36303A3530 05:06:18
  48147018  0000000000000000 ........
  48147020  000000004C425442 BTBL....
  48147028  2E30302E37302E33 3.07.00.
  48147030  00333030302D3030 00-0003.
  48147038  0000000000000000 ........
  48147040  0000000000000000 ........
  48147048  32203133206C754A Jul 31 2
  48147050  0000000000353130 015.....
  48147058  38313A37343A3431 14:47:18
  48147060  0000000000000000 ........
  48147068  0000000000000000 ........
  48147070  0000000000000000 ........
  48147078  0000000000000000 ........
  48147080  0000000000000000 ........
  48147088  0000000000000000 ........
  48147090  0000000000000000 ........
  48147098  0000000000000000 ........
  481470A0  0000000000000000 ........
  481470A8  0000000000000000 ........
  481470B0  0000000000000000 ........
  481470B8  0000000000000000 ........
  481470C0  0000000000000000 ........
  481470C8  0000000000000000 ........
  481470D0  0000000000000000 ........
  481470D8  0000000000000000 ........
  481470E0  0000000000000000 ........
  481470E8  0000000000000000 ........
  481470F0  0000000000000000 ........
  481470F8  0000000000000000 ........
  48147100  0000000000000000 ........
  48147108  0000000000000000 ........
  48147110  0000000000000000 ........
  48147118  0000000000000000 ........
  48147120  0000000000000000 ........
  48147128  0000000000000000 ........
  48147130  0000000000000000 ........
  48147138  0000000000000000 ........
  48147140  0000000000000000 ........
  48147148  0000000000000000 ........
  48147150  0000000000000000 ........
  48147158  0000000000000000 ........
  48147160  0000000000000000 ........
  48147168  0000000000000000 ........
  48147170  0000000000000000 ........
  48147178  0000000000000000 ........
  48147180  0000000000000000 ........
  48147188  0000000000000000 ........
  48147190  0000000000000000 ........
  48147198  0000000000000000 ........

LBRfr0 Dump:
  460F4BA5  FF C8 48 89 44 24 08 48 8B 04 24 48 FF C8 48 89
  460F4BB5  04 24 48 8B 04 24 48 8B 4C 24 08 0F B6 09 88 08
  460F4BC5  EB C1 48 8B 44 24 20 48 83 C4 18 C3 CC CC CC CC
  460F4BD5  CC CC CC CC CC CC CC CC CC CC CC 44 88 44 24 18

RIP Dump:
  460F4BA0  8B 44 24 08 48 FF C8 48 89 44 24 08 48 8B 04 24
  460F4BB0  48 FF C8 48 89 04 24 48 8B 04 24 48 8B 4C 24 08
  460F4BC0  0F B6 09 88 08 EB C1 48 8B 44 24 20 48 83 C4 18
  460F4BD0  C3 CC CC CC CC CC CC CC CC CC CC CC CC CC CC CC
Machine Check Banks:
  MC00  STAT=0000000000000000  ADDR=0000000000000000  MISC=0000000000000000
  MC01  STAT=BD80000000100134  ADDR=0000000046084380  MISC=0000000000000086
  MC02  STAT=0000000000000000  ADDR=0000000000000000  MISC=0000000000000000
  MC03  STAT=0000000000000000  ADDR=0000000000000000  MISC=0000000000000000
  MC04  STAT=0000000000000000  ADDR=0000000000000000  MISC=0000000000000000
  MC05  STAT=0000000000000000  ADDR=0000000000000000  MISC=0000000000000000
  MC06  STAT=0000000000000000  ADDR=0000000000000000  MISC=0000000000000000
  MC07  STAT=0000000000000000  ADDR=0000000000000000  MISC=0000000000000000
  MC08  STAT=0000000000000000  ADDR=0000000046084380  MISC=200400C02FE02086
  MC09  STAT=0000000000000000  ADDR=0000000000000000  MISC=0000000000000000
  MC10  STAT=0000000000000000  ADDR=0000000000000000  MISC=0000000000000000
  MC11  STAT=0000000000000000  ADDR=0000000000000000  MISC=0000000000000000
  MC12  STAT=0000000000000000  ADDR=0000000000000000  MISC=0000000000000000
  MC13  STAT=0000000000000000  ADDR=0000000000000000  MISC=0000000000000000
  MC14  STAT=0000000000000000  ADDR=0000000000000000  MISC=0000000000000000
  MC15  STAT=0000000000000000  ADDR=0000000000000000  MISC=0000000000000000
  MC16  STAT=0000000000000000  ADDR=000000000AA52BC0  MISC=090821EFB025C086
  MC17  STAT=0000000000000000  ADDR=0000000000000000  MISC=0000000000000000
  MC18  STAT=0000000000000000  ADDR=0000000000000000  MISC=0000000000000000
  MC19  STAT=0000000000000000  ADDR=0000000000000000  MISC=0000000000000000


TSC value when exception occurred: 0095F4E57CF2
CPU TSC frequency                : 000095191910 (2,501,450,000)

Log of UEFI DxeCore Events:
  Now-StartTSC Now-EndTSC   TPL  FuncAddr
  FFFFFFFFD1551BBE FFFFFFFFD154B27A  1F  468E4B70 XhciDxe.efi +008B70
  FFFFFFFFD1551FA6 FFFFFFFFD1551D1A  1F  4816AA20 DxeCore.efi +022A20
  FFFFFFFFD17B4D7C FFFFFFFFD17AE3B0  1F  468E4B70 XhciDxe.efi +008B70
  FFFFFFFFD17B514E FFFFFFFFD17B4ED0  1F  4816AA20 DxeCore.efi +022A20
  FFFFFFFFD1A10836 FFFFFFFFD1A09EB2  1F  468E4B70 XhciDxe.efi +008B70
  FFFFFFFFD1A13292 FFFFFFFFD1A13006  1F  4816AA20 DxeCore.efi +022A20
  FFFFFFFFD1C72F30 FFFFFFFFD1C6C504  1F  468E4B70 XhciDxe.efi +008B70
  FFFFFFFFD1C74284 FFFFFFFFD1C73FE2  1F  4816AA20 DxeCore.efi +022A20
  FFFFFFFFD1ED3220 FFFFFFFFD1ECC8B6  1F  468E4B70 XhciDxe.efi +008B70
  FFFFFFFFD1ED4D90 FFFFFFFFD1ED4B1A  1F  4816AA20 DxeCore.efi +022A20
  FFFFFFFFD21381AC FFFFFFFFD21317D8  1F  468E4B70 XhciDxe.efi +008B70
  FFFFFFFFD2138572 FFFFFFFFD2138302  1F  4816AA20 DxeCore.efi +022A20
  FFFFFFFFD239244E FFFFFFFFD23921BA  1F  4737CC60 CpuMpDxe.efi +002C60
  FFFFFFFFD2398EA6 FFFFFFFFD2392634  1F  468E4B70 XhciDxe.efi +008B70
  FFFFFFFFD23993B4 FFFFFFFFD2398FFA  1F  4816AA20 DxeCore.efi +022A20
  FFFFFFFFD25F4A8C FFFFFFFFD25EE186  1F  468E4B70 XhciDxe.efi +008B70
  FFFFFFFFD25F77D0 FFFFFFFFD25F74C8  1F  4816AA20 DxeCore.efi +022A20
  FFFFFFFFD285A800 FFFFFFFFD2853EB0  1F  468E4B70 XhciDxe.efi +008B70
  FFFFFFFFD285ABE2 FFFFFFFFD285A954  1F  4816AA20 DxeCore.efi +022A20
  FFFFFFFFD2ABB7D4 FFFFFFFFD2AB4F16  1F  468E4B70 XhciDxe.efi +008B70
  FFFFFFFFD2ABBB9E FFFFFFFFD2ABB926  1F  4816AA20 DxeCore.efi +022A20
  FFFFFFFFD2D179C8 FFFFFFFFD2D110DA  1F  468E4B70 XhciDxe.efi +008B70
  FFFFFFFFD2D19F6C FFFFFFFFD2D19CFA  1F  4816AA20 DxeCore.efi +022A20
  FFFFFFFFD2F77548 FFFFFFFFD2F70BBC  1F  468E4B70 XhciDxe.efi +008B70
  FFFFFFFFD2F7A950 FFFFFFFFD2F7A6CE  1F  4816AA20 DxeCore.efi +022A20
  FFFFFFFFD31DDF26 FFFFFFFFD31D761E  1F  468E4B70 XhciDxe.efi +008B70
  FFFFFFFFD31DE302 FFFFFFFFD31DE086  1F  4816AA20 DxeCore.efi +022A20
  FFFFFFFFD343ECB0 FFFFFFFFD3438354  1F  468E4B70 XhciDxe.efi +008B70
  FFFFFFFFD343F0A8 FFFFFFFFD343EE10  1F  4816AA20 DxeCore.efi +022A20
  FFFFFFFFD369F888 FFFFFFFFD3698EB0  1F  468E4B70 XhciDxe.efi +008B70
  FFFFFFFFD369FC62 FFFFFFFFD369F9E0  1F  4816AA20 DxeCore.efi +022A20
  FFFFFFFFD39005D4 FFFFFFFFD38F9C7C  1F  468E4B70 XhciDxe.efi +008B70
  FFFFFFFFD39009CA FFFFFFFFD390072C  1F  4816AA20 DxeCore.efi +022A20
  FFFFFFFFD3B4B73C FFFFFFFFD3B4B4DE  1F  4737CC60 CpuMpDxe.efi +002C60
  FFFFFFFFD3B52198 FFFFFFFFD3B4B8F8  1F  468E4B70 XhciDxe.efi +008B70
  FFFFFFFFD3B57282 FFFFFFFFD3B521C0  1F  469F9C70 TerminalDxe.efi +004C70
  FFFFFFFFD3B61098 FFFFFFFFD3B572AA  1F  469F9C70 TerminalDxe.efi +004C70
  FFFFFFFFD3B61844 FFFFFFFFD3B611EC  1F  4816AA20 DxeCore.efi +022A20
  FFFFFFFFD3DBC682 FFFFFFFFD3DB5D54  1F  468E4B70 XhciDxe.efi +008B70
  FFFFFFFFD3DBFB28 FFFFFFFFD3DBF81E  1F  4816AA20 DxeCore.efi +022A20
  FFFFFFFFD4022EC2 FFFFFFFFD401C62A  1F  468E4B70 XhciDxe.efi +008B70
  FFFFFFFFD40232B0 FFFFFFFFD4023018  1F  4816AA20 DxeCore.efi +022A20
  FFFFFFFFD427E346 FFFFFFFFD4277A4A  1F  468E4B70 XhciDxe.efi +008B70
  FFFFFFFFD4281634 FFFFFFFFD428139A  1F  4816AA20 DxeCore.efi +022A20
  FFFFFFFFD44E475A FFFFFFFFD44DDDC8  1F  468E4B70 XhciDxe.efi +008B70
  FFFFFFFFD44E4B34 FFFFFFFFD44E48AE  1F  4816AA20 DxeCore.efi +022A20
  FFFFFFFFD474562A FFFFFFFFD473ED36  1F  468E4B70 XhciDxe.efi +008B70
  FFFFFFFFD47459F6 FFFFFFFFD474577E  1F  4816AA20 DxeCore.efi +022A20
  FFFFFFFFD49A60C6 FFFFFFFFD499F728  1F  468E4B70 XhciDxe.efi +008B70
  FFFFFFFFD49A64AE FFFFFFFFD49A6226  1F  4816AA20 DxeCore.efi +022A20
  FFFFFFFFD4C07066 FFFFFFFFD4C007B2  1F  468E4B70 XhciDxe.efi +008B70
  FFFFFFFFD4C07430 FFFFFFFFD4C071BA  1F  4816AA20 DxeCore.efi +022A20
  FFFFFFFFD4E6443E FFFFFFFFD4E5DB7E  1F  468E4B70 XhciDxe.efi +008B70
  FFFFFFFFD4E657D0 FFFFFFFFD4E65564  1F  4816AA20 DxeCore.efi +022A20
  FFFFFFFFD50C8A9A FFFFFFFFD50C212E  1F  468E4B70 XhciDxe.efi +008B70
  FFFFFFFFD50C8E86 FFFFFFFFD50C8BE8  1F  4816AA20 DxeCore.efi +022A20
  FFFFFFFFD5322C00 FFFFFFFFD53229AE  1F  4737CC60 CpuMpDxe.efi +002C60
  FFFFFFFFD53296DC FFFFFFFFD5322DE6  1F  468E4B70 XhciDxe.efi +008B70
  FFFFFFFFD5329BEE FFFFFFFFD532982E  1F  4816AA20 DxeCore.efi +022A20
  FFFFFFFFD558A474 FFFFFFFFD5583B5A  1F  468E4B70 XhciDxe.efi +008B70
  FFFFFFFFD558A8CA FFFFFFFFD558A5C2  1F  4816AA20 DxeCore.efi +022A20
  FFFFFFFFD57EB24A FFFFFFFFD57E48D6  1F  468E4B70 XhciDxe.efi +008B70
  FFFFFFFFD57EB628 FFFFFFFFD57EB3A6  1F  4816AA20 DxeCore.efi +022A20
  FFFFFFFFD5A4BF7E FFFFFFFFD5A456E8  1F  468E4B70 XhciDxe.efi +008B70
Log of UEFI DXE drivers that were started:
  Now-StartTSC Now-EndTSC   FuncAddr
  0000E64E6EFC XXXXXXXXXXXX 462EA2DC Unknown(ngkkh) +0102DC Avago EFI SAS Driver
  0000E7315D16 0000E72F7EF2 4694FA60 PciBusDxe.efi +000A60
  000103A01AD2 0001039E4486 4694FA60 PciBusDxe.efi +000A60
  0001081D2A64 0001081B4E3A 4694FA60 PciBusDxe.efi +000A60
  0001135DE4A8 0001135BFF82 4694FA60 PciBusDxe.efi +000A60
  000121E1CD4A 000121DFADBA 4694FA60 PciBusDxe.efi +000A60
  0002847479B6 000142AF1DA0 46350840 cxundi.efi +05F840
  0003C7A0FBD8 0002854982E6 463DE840 cxundi.efi +05F840
Log of UEFI DXE drivers that were Stopped:
  Now-StopTSC Now-EndTSC   FuncAddr
  002354373E62 00235435E17A 46947CB0 IsaAcpi.efi +000CB0
  002354381558 002354377246 4693BFF0 IsaBusDxe.efi +000FF0
  0023543C7EA4 00235438156E 46860BF0 IsaSerialDxe.efi +001BF0
  002355282192 0023543C7EF0 469F7320 TerminalDxe.efi +002320
  00235528F3C6 0023552821CA 46A091A0 ConSplitterDxe.efi +0021A0
  0023552A0BCA 00235528F402 46A091A0 ConSplitterDxe.efi +0021A0
  0023552AC36C 0023552AC2FE 46A09110 ConSplitterDxe.efi +002110
  0023552B99A6 0023552AC3A0 46A09110 ConSplitterDxe.efi +002110

EFI Debug Support Table UpdateStatus: 2
EFI Debug Support Table TableSize   : 263
  48148000 00068000 DxeCore.efi
  4BAF6000 000082C0 PcdDxe.efi
  5DDF1000 0000E000 RegAccessDxe.efi
  5DDE2000 0000F000 ReportStatusCodeRouterRuntimeDxe.efi
  4BAEC000 000090A0 SectionExtractionDxe.efi
  4BADA000 00011160 PlatformEarlyDxe.efi
  5DDD6000 0000C000 CpuIoDxe.efi
  4BAD3000 00006E60 CpuIo2Dxe.efi
  4BACB000 00007FE0 DataHubDxe.efi
  4BAC4000 000067E0 Legacy8259.efi
  4BAB3000 000103E0 CpuArchDxe.efi
  4BAA4000 00007680 Metronome.efi
  5DDC8000 0000E000 RuntimeDxe.efi
  4BA9D000 00006E80 DpcDxe.efi
  4BA93000 00009240 RngDxe.efi
  4BA79000 00019B20 HstiIhvProviderDxe.efi
  4BA71000 00007D00 HttpUtilitiesDxe.efi
  4BA61000 0000F040 AcpiTableDxe.efi
  4BA4C000 000143A0 DevicePathDxe.efi
  4BA3F000 0000CF00 EbcDxe.efi
  4BA34000 0000AA00 PchSmbusDxe.efi
  4BA2C000 000077C0 LegacyInterrupt.efi
  5DDBB000 0000D000 SmmControl.efi
  4BA20000 0000B520 PchSerialGpio.efi
  4BA18000 00007CE0 WdtDxe.efi
  4BA11000 000068E0 IncompatiblePciDeviceSupport.efi
  4BA06000 0000A360 SmbiosDxe.efi
  4B9FF000 00006E60 SmmCommunicationBufferDxe.efi
  4B9F4000 0000A800 S3SaveStateDxe.efi
  4B9ED000 00006600 PciHotPlug.efi
  5DDAF000 0000C000 NvdimmCommon.efi
  5DDA3000 0000C000 NvdimmSmbusCommon.efi
  4B9E5000 00007940 Idrac9ChipsetDxe.efi
  4B9DE000 00006640 DellIdracInterface.efi
  4B9D7000 00006CC0 DellIpmiDebug2.efi
  4B9D0000 00006920 DellHiiBrowserCapability.efi
  4B9C9000 00006A60 DellPerformanceMeasurementDxe.efi
  4B9BC000 0000C040 JsonParserDxe.efi
  4B9B4000 00007580 DellCmosAccessDxe.efi
  4B9AC000 00007240 DellVamDxe.efi
  4B9A3000 000085C0 DellSmaDebug.efi
  4B998000 0000A060 DellFruAccessDxe.efi
  4B990000 000072E0 IdracPsuBootStatusDxe.efi
  4B989000 00006760 DpiNvdimmDxe.efi
  4B938000 00050D60 DellRSOD.efi
  4B908000 0002F460 HiiDatabase.efi
  4B8E1000 00026CC0 DellImageDrawDxe.efi
  4B870000 00070E40 Gdriver.efi
  4B81A000 00055180 MRE.efi
  4B813000 00006FC0 DellNmiButtonDxe.efi
  4B80C000 00007000 DpiFlashData2Dxe.efi
  4B7F6000 00015DE0 DpiPci2Dxe.efi
  4B7EF000 00006F00 DpiCpuDxe.efi
  4B7E4000 0000A440 DpiPercDxe.efi
  4B7DD000 00006AC0 DpiMemoryDxe.efi
  4B7D4000 000083A0 DellSystemPortInfoDxe.efi
  4B7CD000 00006400 DpiVrDxe.efi
  4B7C2000 0000A5A0 DpiNvmeDxe.efi
  4B7BB000 00006840 SmbiosTableListDxe.efi
  4B7B4000 00006D80 DellPlatformStaticOverridesDxe.efi
  5DD98000 0000B000 DellOemHookStatusCodeHandlerDxe.efi
  4B7AA000 00009AE0 HstiPlatformDxe.efi
  4B7A3000 00006280 DataHubStdErrDxe.efi
  4B79A000 00008F20 GenericMemoryTestDxe.efi
  4B792000 00007840 SmartTimer.efi
  4B788000 00009BC0 FirmwarePerformanceDxe.efi
  4B781000 00006400 DellPercDxe.efi
  4B774000 0000C480 PersonalityModuleDataDxe.efi
  4B76B000 00008DC0 PersonalityModuleBrandingDxe.efi
  480BA000 0006D300 DellHiiBrowser.efi
  4B701000 00069540 DellHiiFormsProcessor2Dxe.efi
  4804A000 0006F320 SystemBoardDxe.efi
  48041000 00008340 PlatformType.efi
  4803A000 00006900 WatchdogTimer.efi
  48028000 00011620 DriverHealthManagerDxe.efi
  48019000 0000E2E0 TlsAuthConfigDxe.efi
  48004000 000146A0 PciPlatform.efi
  47DDE000 00021240 DisplayEngine.efi
  47DC3000 0001A3E0 PlatDriOverrideDxe.efi
  47DBA000 00008F80 DellHiiDefaults.efi
  47DAC000 0000D6E0 DellHiiConfigDxe.efi
  47D62000 000496A0 DellHiiFormDisplay.efi
  47D4F000 00012120 DpiPlatformIdDxe.efi
  47D32000 0001CAA0 PciHostBridge.efi
  5DD87000 00011000 CpuCsrAccess.efi
  47D2A000 000070A0 DellChipsetInfoDxe.efi
  47D1F000 0000A580 DpiClocksDxe.efi
  47D0D000 00011AE0 DpiSataDxe.efi
  47CFB000 00011F40 DpiUsbDxe.efi
  47CEC000 0000ECC0 DpiNdcDxe.efi
  47CD3000 00018AC0 DellNvramDefaultAttributeDxe.efi
  47CCB000 000075C0 DellPlatformAttributeDefaultsDxe.efi
  5DD78000 0000F000 PchSpiRuntime.efi
  5DD6A000 0000E000 PchResetRuntime.efi
  47CC3000 00007A20 SmmAccess.efi
  47CBC000 00006340 DellClocksDxe.efi
  47CB3000 000082C0 DellPreviousBootErrorDxe.efi
  47CA0000 00012A40 SbServiceProtocolDxe.efi
  47C99000 000068C0 DellPlatformNvramDefaultsTableDxe.efi
  5DD59000 00011000 FwBlockService.efi
  5DD45000 00014000 PiSmmIpl.efi
  47C90000 00008140 DellSpiFvbServicesDxe.efi
  47C89000 000066E0 DpiTpmDxe.efi
  5DD35000 00010000 VariableSmmRuntimeDxe.efi
  47C7F000 000098E0 DellNvramAttributeSmmDxe.efi
  47C76000 000086A0 DellSpiAccessDxe.efi
  47C60000 000155C0 PlatformCpuPolicy.efi
  47C52000 0000DAA0 LpcPlatform.efi
  47944000 000BB940 SecurityStubDxe.efi
  47C4A000 000070A0 LegacyRegion2.efi
  47C16000 00033E40 PchInitDxe.efi
  5DD21000 00014000 PlatformReset.efi
  47C04000 00011200 AcpiVTD.efi
  47931000 00012040 HeciInitDxe.efi
  47913000 0001D340 DellFirmwareManagementDxe.efi
  478FE000 00014720 DellPciDevice2Dxe.efi
  478F5000 00008040 DellManufacturingModeDxe.efi
  478E9000 0000B7E0 DellAciMDxe.efi
  478E0000 00008520 DellIpmi.efi
  478CC000 00013880 DellMainMenuSetupDxe.efi
  478A0000 0002BE40 DellBootManagerDxe.efi
  47884000 0001B400 DellHiiInternalSetup.efi
  47879000 0000AF60 DellHiiBrowserData.efi
  4785C000 0001CFE0 DellSmaJobProcess.efi
  4784E000 0000DEE0 TlvInventoryDxe.efi
  47846000 00007160 DellLegacyTableMemoryDxe.efi
  4783E000 00007E20 DellPasswordInterfaceDxe.efi
  47810000 0002D5E0 DellPostErrorMessageDxe.efi
  475E7000 00016000 DellBiosInterfaceDxe.efi
  47804000 0000BF00 HiiServiceTag.efi
  475DD000 00009D00 DellIpmiResetToDefaultsDxe.efi
  475C6000 00016900 DellFileExplorerDxe.efi
  475AD000 00018D40 DellPmDataAcquisitionDxe.efi
  4758B000 00021500 DellSmbiosTableBuildDxe.efi
  47580000 0000A9A0 DellRci2BuildTable.efi
  47579000 000065C0 DellWmiAcpiBuffer.efi
  47570000 00008420 DellBiosDataWipe.efi
  474DA000 00095380 DellCpldFirmwareManagementDxe.efi
  474D0000 00009140 DellBootToDiagsDxe.efi
  474BD000 00012300 DellEmbSata.efi
  4742D000 0008F120 DellIdracSecureBootMgmt.efi
  47423000 00009DC0 DellSetBayAndSlotDxe.efi
  5DD10000 00011000 DellNvdimmDxe.efi
  47411000 000118E0 DellAepSecurityDxe.efi
  473FF000 00011CA0 DellRaidFmpDxe.efi
  473F6000 00008CC0 DellCapsuleUpdateDxe.efi
  473ED000 000084A0 DellHiiWorkloadProfile.efi
  473DF000 0000D6E0 DellChipsetMemInfoDxe.efi
  473D8000 00006E20 DpiIdrac.efi
  473D1000 000064C0 DpiBdsDxe.efi
  473B5000 0001BD60 DellEarlyVideoSplashDxe.efi
  4737A000 0003A580 CpuMpDxe.efi
  47373000 00006860 DellMemoryInterfaceMemoryDxe.efi
  4736B000 00007EE0 DellIdracInitialize.efi
  4735B000 0000F720 DellDlst2.efi
  4734F000 0000B820 DellNdcDxe.efi
  47348000 000069A0 DellSecurityLayerDxe.efi
  4732F000 00018A40 DellIdrac.efi
  47313000 0001B500 DellLcProvisioning.efi
  4730B000 00007580 DellSmmCommBufferTableBuildDxe.efi
  47301000 00009420 DellRipsDxe.efi
  472F2000 0000E2E0 DellIdracSma.efi
  472E1000 00010300 DpiCpldDxe.efi
  5DCFF000 00011000 PcRtc.efi
  5DCF3000 0000C000 MonotonicCounterRuntimeDxe.efi
  5DCD9000 0001A000 CapsuleRuntimeDxe.efi
  472D8000 00008CA0 EsrtDxe.efi
  472C9000 0000E480 SoftSkuStatus.efi
  472B9000 0000FA40 IsPlatformSupportWhea.efi
  472A3000 00015B60 MePolicyInitDxe.efi
  47279000 0002A000 IioInit.efi
  47251000 00027DA0 DellDeviceLocation.efi
  4723E000 00012B40 DellCpuInterfaceDxe.efi
  47236000 00008000 DellPmQueryDxe.efi
  47225000 00010640 DellPmDataStorageDxe.efi
  5DCCC000 0000D000 CallingInterface.efi
  471F2000 000329A0 DellTpm.efi
  471D9000 00018C00 DellProcessCapsuleDxe.efi
  4711E000 000BA620 HiiSecureBootCustom.efi
  47117000 00006BC0 UFSDxe.efi
  470FE000 00018FC0 DellNvdimmChipsetDxe.efi
  470F1000 0000C1C0 LegacyBiosPlatform.efi
  470E7000 00009680 WheaSupport.efi
  47003000 000E4000 CrystalRidge.efi
  46FE6000 0001CFC0 SpsDxe.efi
  46FDC000 00009A80 DellTurboEnablePerCore.efi
  46FD1000 0000A6C0 PerformanceControlDxe.efi
  46FAC000 00024800 LegacyBiosDxe.efi
  46F9E000 0000DF40 DellPciElogDxe.efi
  46F96000 00007AA0 DpiCpuVrDxe.efi
  46F86000 0000F040 PpmInitialize.efi
  46F7F000 00006CC0 LegacyInterruptHookDxe.efi
  46F77000 00007BE0 LegacyBiosReverseThunk.efi
  46E93000 000E4000 JedecNvDimm.efi
  46E88000 0000AA00 DellUsbCsmSwitch.efi
  46E77000 00010620 DellTxt.efi
  46E41000 00035780 AcpiPlatform.efi
  46DA1000 0009F9A0 DellBdsDxe.efi
  46D45000 0005B140 Unknown(stgsr) Intel(R) RSTe 5.5.5.1005 SATA Driver
  46CE9000 0005B140 Unknown(stgsr) Intel(R) RSTe 5.5.5.1005 sSATA Driver
  46CDC000 0000CAC0 S3NvramSave.efi
  46CCB000 000100A0 ReserveMem.efi
  46CBD000 0000D940 ArpDxe.efi
  46CA7000 00015600 Dhcp4Dxe.efi
  46C83000 00023BA0 Ip4Dxe.efi
  46C6D000 000157E0 MnpDxe.efi
  46C5A000 00012FC0 Mtftp4Dxe.efi
  46C49000 00010C60 SnpDxe.efi
  46C36000 00012FA0 Udp4Dxe.efi
  46C1F000 00016180 Dhcp6Dxe.efi
  46BF5000 000294C0 Ip6Dxe.efi
  46BE0000 00014160 Mtftp6Dxe.efi
  46BBC000 00023240 TcpDxe.efi
  46BA8000 00013180 Udp6Dxe.efi
  46B88000 0001FE80 UefiPxeBcDxe.efi
  46A80000 001071E0 TlsDxe.efi
  46A64000 0001B060 HttpDxe.efi
  46A49000 0001A660 HttpBootDxe.efi
  46A31000 00017960 DnsDxe.efi
  46A22000 0000EC80 BiosVideoDxe.efi
  46A18000 00009A20 ConPlatformDxe.efi
  46A07000 00010F20 ConSplitterDxe.efi
  469F5000 00011F60 TerminalDxe.efi
  469E6000 0000EBE0 PartitionDxe.efi
  469D8000 0000DF80 UdfDxe.efi
  469CE000 00009560 VgaClassDxe.efi
  469C3000 0000AF60 DiskIoDxe.efi
  469AE000 000149A0 Fat.efi
  469A3000 0000A7E0 SataController.efi
  46993000 0000F380 AtaBusDxe.efi
  4697C000 00016D40 AtaAtapiPassThruDxe.efi
  4694F000 0002CEE0 PciBusDxe.efi
  46947000 00007900 IsaAcpi.efi
  4693B000 0000B240 IsaBusDxe.efi
  46934000 00006F00 EnglishDxe.efi
  46928000 0000B060 ScsiBus.efi
  46913000 00014460 ScsiDisk.efi
  46903000 0000F400 UsbKbDxe.efi
  468F8000 0000AE00 UsbMouseDxe.efi
  468DC000 0001BD80 XhciDxe.efi
  468C9000 00012240 EhciDxe.efi
  468BB000 0000DFE0 OhciDxe.efi
  468A6000 00014140 UsbBusDxe.efi
  46898000 0000DB80 UsbMassStorageDxe.efi
  46889000 0000E140 DellCdDvdFileSystem.efi
  46879000 0000F760 BootOptionCacheDxe.efi
  4686D000 0000B180 DellUsbMouseAbsolutePointerDxe.efi
  4685F000 0000DD20 IsaSerialDxe.efi
  46827000 00038000 NvmExpressDxe.efi
  467BA000 0006CB00 DellHiiTextDisplay.efi
  46749000 000706E0 DellHiiGraphicDisplay.efi
  46711000 00037DA0 RegularExpressionDxe.efi
  46708000 000080A0 DellXMLDrv1.efi
  466E6000 00021660 DellHiiConverter.efi
  465DD000 001085E0 iDRAC_Hii.efi
  465CF000 0000DF60 Unknown(gnjgs) G200eW Matrox Graphics UEFI Driver
  464A1000 0012DBC0 RAIDCore.efi
  46457000 00049800 b57undi.efi
  4640D000 00049800 b57undi.efi
  4637F000 0008D2C0 cxundi.efi
  462F1000 0008D2C0 cxundi.efi
  462DA000 00016400 Unknown(ngkkh) Avago EFI SAS Driver
  45EFE000 003DB860 Unknown(ttwgg)

Log Size: 41,794

Not sure where this comes from. But es2021 doesn't go past that.

Also, I have seen several errors logged with DIMM_A5 module:

Record:      136
Date/Time:   01/11/2020 01:01:50
Source:      system
Severity:    Critical
Description: Multi-bit memory errors detected on a memory device at location(s) DIMM_A5.

Record:      106
Date/Time:   01/09/2020 23:21:01
Source:      system
Severity:    Critical
Description: Correctable memory error logging disabled for a memory device at location DIMM_A5.

Record:      47
Date/Time:   01/08/2020 15:54:25
Source:      system
Severity:    Critical
Description: Multi-bit memory errors detected on a memory device at location(s) DIMM_A5.

I have tried es2024 and setting its IP manually and it seems that it indeed cannot reach the network. However, I do see the interfaces, but they don't seem to be having link

~ # ip address | grep state
~ #
~ # ip address
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eno3: <BROADCAST,MULTICAST> mtu 1500 qdisc mq qlen 1000
    link/ether b0:26:28:f5:2c:04 brd ff:ff:ff:ff:ff:ff
3: eno1np0: <BROADCAST,MULTICAST> mtu 1500 qdisc mq qlen 1000
    link/ether b0:26:28:f5:2c:06 brd ff:ff:ff:ff:ff:ff
    inet 10.192.0.158/22 brd 10.192.3.255 scope global eno1np0
       valid_lft forever preferred_lft forever
4: eno4: <BROADCAST,MULTICAST> mtu 1500 qdisc mq qlen 1000
    link/ether b0:26:28:f5:2c:05 brd ff:ff:ff:ff:ff:ff
5: eno2np1: <BROADCAST,MULTICAST> mtu 1500 qdisc mq qlen 1000
    link/ether b0:26:28:f5:2c:07 brd ff:ff:ff:ff:ff:ff

 # route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
~ # route add default gw 10.192.0.1
route: SIOCADDRT: Network is unreachable

~ # lspci | grep net
01:00.0 Ethernet controller: Broadcom Limited NetXtreme BCM5720 Gigabit Ethernet PCIe
01:00.1 Ethernet controller: Broadcom Limited NetXtreme BCM5720 Gigabit Ethernet PCIe
18:00.0 Ethernet controller: Broadcom Limited BCM57412 NetXtreme-E 10Gb RDMA Ethernet Controller (rev 01)
18:00.1 Ethernet controller: Broadcom Limited BCM57412 NetXtreme-E 10Gb RDMA Ethernet Controller (rev 01)

@Papaul if you visually check the network card, do they have link? does the switch see them being up?

@Marostegui

papaul@asw-a-codfw> show interfaces ge-6/0/12 descriptions    
Interface       Admin Link Description
ge-6/0/12       up    up   es2024

for es2021 i know already about it

@Marostegui

papaul@asw-a-codfw> show interfaces ge-6/0/12 descriptions    
Interface       Admin Link Description
ge-6/0/12       up    up   es2024

for es2021 i know already about it

Is es2024 also in the correct VLAN?

@Marostegui

Logical          Vlan          TAG     MAC         STP         Logical           Tagging 
interface        members               limit       state       interface flags  
ge-6/0/12.0                            294912                                     untagged   
                 private1-a-codfw 2017 294912      Forwarding                     untagged

I just saw this:

[35393.835622] tg3 0000:01:00.0 eno3: Link is up at 1000 Mbps, full duplex
[35393.835634] tg3 0000:01:00.0 eno3: Flow control is off for TX and off for RX
[35393.835636] tg3 0000:01:00.0 eno3: EEE is disabled
[35393.835660] IPv6: ADDRCONF(NETDEV_CHANGE): eno3: link becomes ready
[35397.323562] IPv6: ADDRCONF(NETDEV_UP): eno1np0: link is not ready

@Marostegui switch didn't learn any MAC address on that interface

papaul@asw-a-codfw> show ethernet-switching table interface ge-6/0/12    

MAC database for interface ge-6/0/12

MAC database for interface ge-6/0/12.0

You are not able to see any MAC addresses for es2024?

correct, on the switch side

I have unloaded and then loaded again bnxt_en kernel module and I can see the main iface disappearing and then coming back. So I am not sure this is OS related, I believe there must be something weird with the switch/vlan or something?

[36855.676818] Broadcom NetXtreme-C/E driver bnxt_en v1.9.2
[36855.688230] bnxt_en 0000:18:00.0 eth0: Broadcom BCM57412 NetXtreme-E 10Gb Ethernet found at mem 9da10000, node addr b0:26:28:f5:2c:06
[36855.688241] bnxt_en 0000:18:00.0: 63.008 Gb/s available PCIe bandwidth (8 GT/s x8 link)
[36855.688483] bnxt_en 0000:18:00.0 eno1np0: renamed from eth0
[36855.713617] bnxt_en 0000:18:00.1 eth0: Broadcom BCM57412 NetXtreme-E 10Gb Ethernet found at mem 9da00000, node addr b0:26:28:f5:2c:07
[36855.713627] bnxt_en 0000:18:00.1: 63.008 Gb/s available PCIe bandwidth (8 GT/s x8 link)
[36855.715133] bnxt_en 0000:18:00.1 eno2np1: renamed from eth0

@Marostegui i double check the switch configuration for both es2020 and es2024 and the DNS files from https://gerrit.wikimedia.org/r/#/c/operations/dns/+/563323/ everything looks good. I did another test on es2021 when the server boots at the step of PXE boot the switch learned the MAC address

Vlan                MAC                 MAC         Age    Logical
    name                address             flags              interface
    private1-a-codfw    b0:26:28:f5:28:02   D             -   ge-3/0/27.0      
``
an just after the installation begins the MAC address goes away

@Marostegui I will focus more on troubleshooting this on the NIC level on Monday. Since the 1GB and 10GB interfaces are on the same card, I will use the 10GB interface over the 1GB interface and see if if that helps. Have a great week end.

Sounds good @Papaul! Thank you a lot.
Have a good one!!

Marostegui moved this task from Triage to In progress on the DBA board.Jan 12 2020, 6:57 AM

@Marostegui I will focus more on troubleshooting this on the NIC level on Monday. Since the 1GB and 10GB interfaces are on the same card, I will use the 10GB interface over the 1GB interface and see if if that helps. Have a great week end.

@Papaul we bought this hosts with 10G NICs just in case, but we can try with 1G at the moment (like the other existing hosts use). We are nowhere near NIC saturation on those hosts, so 1G is fine. Arzhel suggests that maybe there is an option on the BIOS to "disable" the 10G part of those NICs and leave them with just 1G enabled?

I have tried to look thru the BIOS to find a way to disable the 10G capability but I have found nothing.

On the installer shell I have tried removing the driver for the 10G card and leave only 1G which seems to work:

~ # modprobe -r bnxt_en
~ # ip address
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
3: eno3: <BROADCAST,MULTICAST> mtu 1500 qdisc mq qlen 1000
    link/ether b0:26:28:f5:2c:04 brd ff:ff:ff:ff:ff:ff
5: eno4: <BROADCAST,MULTICAST> mtu 1500 qdisc mq qlen 1000
    link/ether b0:26:28:f5:2c:05 brd ff:ff:ff:ff:ff:ff

After that, I have given eno3 and IP via network/interfaces, however the installer keeps wanting to use eno1np0 which doesn't appear there anymore as it is unloaded after I removed the module, so the network keeps failing as eno1np0 doesn't exist.
@MoritzMuehlenhoff any idea where that is hardcoded?

For a test maybe disable Puppet on the install* servers and add

d-i netcfg/choose_interface select eno3

to the buster.cfg config file. If that fixes it, we can figure out a good way going forward.

Thanks @MoritzMuehlenhoff - I have merged https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/563975/ to make sure we install (or test) es2024 with stretch, as ideally we'll install stretch on those hosts.
Buster comes with 10.4 and we are not ready for that yet. There is no 10.1 for buster.

So I am going to modify stretch.cfg

The hack didn't work. It keeps choosing a different NIC than eno3, which unfortunately looks like the 10G one.
I tried a different approach which was uninstalling the kernel module BEFORE running the network check, and then I generated the interfaces file manually, so it would only pick eno3, but the installer now looks for eno1 which doesn't exist, as I uninstalled the kernel module bnxt_en.

~ # ip address
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
4: eno3: <BROADCAST,MULTICAST> mtu 1500 qdisc mq qlen 1000
    link/ether b0:26:28:f5:2c:04 brd ff:ff:ff:ff:ff:ff
5: eno4: <BROADCAST,MULTICAST> mtu 1500 qdisc mq qlen 1000
    link/ether b0:26:28:f5:2c:05 brd ff:ff:ff:ff:ff:ff
~ # cat /etc/network/interfaces
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

source /etc/network/interfaces.d/*

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
allow-hotplug eno1
iface eno1 inet static
        address 10.192.0.158/22
        gateway 10.192.0.1
        # dns-* options are implemented by the resolvconf package, if installed
        dns-nameservers 10.3.0.1
        dns-search unassigned

:(

Last test done with @ayounsi:
I have removed the drivers tg3 and bnxt_en from the OS:

/bin # ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
/bin # ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

However they still show as UP on the switch

Some more notes.
The only way to ping the gateway seems to be using the 1G ports, and making sure the 10G aren't there.
The 1G ifaces are controlled by tg3 module, so I have removed both, 1G and 10G drivers and then only loaded 1G one:

~ # modprobe -r bnxt_en
~ # modprobe -r tg3
~ # modprobe tg3
~ # ip addr add 10.192.0.158/22 dev eno3
~ # ip link set eno3 up
~ # ping 10.192.0.1
PING 10.192.0.1 (10.192.0.1): 56 data bytes
64 bytes from 10.192.0.1: seq=0 ttl=64 time=1011.309 ms
64 bytes from 10.192.0.1: seq=1 ttl=64 time=11.212 ms
64 bytes from 10.192.0.1: seq=2 ttl=64 time=3.554 ms

The problem is that the installation keeps waiting for eno1 or eno2 which seem to be the 10G interfaces, and ignores the fact that eno3 is the one with link :(
I have been trying to rename the interfaces but I have not been able to do so, so far, as it is a Busybox shell.

I guess if we can force the installation to only consider eno3 it might work?

The 10G card is identical to what we have running fine on Stretch in e.g. ms-be2050 and I also validated there are no missing firmware packages etc, this seems entirely fine from the Linux end. Have we tried plugging the server into a different switch port or replaced the cable?

Thanks @MoritzMuehlenhoff for your tests.

So recap:

  • this host has 1G and 10G: we bought the 10G because once we start doing backups of these hosts, it will be good to have more speed
  • Currently this host has only a cable on the 1G port, not on the 10G
  • The installation keeps trying to install via the 10G port, which doesn't have link
  • The only way to get a link on the 1G port is to unload the two network drivers and load only the 1G one. However, the installation keeps looking for 10G NICs, which aren't even loaded.

Ideas?

  • Keep trying to see why d-i netcfg/choose_interface select eno3 doesn't work
  • Enable to 10G even though it will go to a 1G switch port? Is that even possible? @ayounsi @Papaul?

For what is worth, exactly the same behaviour is happening on es2020.
And what was done at T242481#5797643 for es2024 also works and the 1G NIC is able to ping its gateway.

No link on any of them:

~ # ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eno1: <BROADCAST,MULTICAST> mtu 1500 qdisc mq qlen 1000
    link/ether b0:26:28:f5:28:04 brd ff:ff:ff:ff:ff:ff
3: eno2d1: <BROADCAST,MULTICAST> mtu 1500 qdisc mq qlen 1000
    link/ether b0:26:28:f5:28:05 brd ff:ff:ff:ff:ff:ff
4: eno3: <BROADCAST,MULTICAST> mtu 1500 qdisc mq qlen 1000
    link/ether b0:26:28:f5:28:02 brd ff:ff:ff:ff:ff:ff
5: eno4: <BROADCAST,MULTICAST> mtu 1500 qdisc mq qlen 1000
    link/ether b0:26:28:f5:28:03 brd ff:ff:ff:ff:ff:ff

Forcing them to come up:

~ # ip link set eno1 up
~ # ip link set eno2d1 up
~ # ip link set eno3 up
~ # ip link set eno4 up

And they do come up:

~ # ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eno1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq qlen 1000
    link/ether b0:26:28:f5:28:04 brd ff:ff:ff:ff:ff:ff
3: eno2d1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq qlen 1000
    link/ether b0:26:28:f5:28:05 brd ff:ff:ff:ff:ff:ff
4: eno3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq qlen 1000
    link/ether b0:26:28:f5:28:02 brd ff:ff:ff:ff:ff:ff
    inet6 2620:0:860:101:b226:28ff:fef5:2802/64 scope global dynamic
       valid_lft 2591981sec preferred_lft 604781sec
    inet6 fe80::b226:28ff:fef5:2802/64 scope link
       valid_lft forever preferred_lft forever
5: eno4: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq qlen 1000
    link/ether b0:26:28:f5:28:03 brd ff:ff:ff:ff:ff:ff

However, they yet don't get an IP from DHCP and installation fails and they seem down again:

~ # ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eno1: <BROADCAST,MULTICAST> mtu 1500 qdisc mq qlen 1000
    link/ether b0:26:28:f5:28:04 brd ff:ff:ff:ff:ff:ff
3: eno2d1: <BROADCAST,MULTICAST> mtu 1500 qdisc mq qlen 1000
    link/ether b0:26:28:f5:28:05 brd ff:ff:ff:ff:ff:ff
4: eno3: <BROADCAST,MULTICAST> mtu 1500 qdisc mq qlen 1000
    link/ether b0:26:28:f5:28:02 brd ff:ff:ff:ff:ff:ff
5: eno4: <BROADCAST,MULTICAST> mtu 1500 qdisc mq qlen 1000
    link/ether b0:26:28:f5:28:03 brd ff:ff:ff:ff:ff:ff

I force them back up:

~ # ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eno1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq qlen 1000
    link/ether b0:26:28:f5:28:04 brd ff:ff:ff:ff:ff:ff
3: eno2d1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq qlen 1000
    link/ether b0:26:28:f5:28:05 brd ff:ff:ff:ff:ff:ff
4: eno3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq qlen 1000
    link/ether b0:26:28:f5:28:02 brd ff:ff:ff:ff:ff:ff
    inet6 2620:0:860:101:b226:28ff:fef5:2802/64 scope global dynamic
       valid_lft 2591998sec preferred_lft 604798sec
    inet6 fe80::b226:28ff:fef5:2802/64 scope link
       valid_lft forever preferred_lft forever
5: eno4: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq qlen 1000
    link/ether b0:26:28:f5:28:03 brd ff:ff:ff:ff:ff:ff

I try to assign an IP to eno3 (1G NIC) and I can successfully ping its gw:

~ # ip addr add 10.192.0.157/22 dev eno3
~ # ip link set eno3 up
~ # ping 10.192.0.1
PING 10.192.0.1 (10.192.0.1): 56 data bytes
64 bytes from 10.192.0.1: seq=0 ttl=64 time=0.896 ms
64 bytes from 10.192.0.1: seq=1 ttl=64 time=0.407 ms

I go back to the installation menu and if fails, as it tries to use eno1, which doesn't have an IP.
I give eno1 the same IP and it of course fails

2: eno1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq qlen 1000
    link/ether b0:26:28:f5:28:04 brd ff:ff:ff:ff:ff:ff
    inet 10.192.0.157/22 brd 10.192.3.255 scope global eno1
       valid_lft forever preferred_lft forever


~ # route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         10.192.0.1      0.0.0.0         UG    0      0        0 eno1
10.192.0.0      0.0.0.0         255.255.252.0   U     0      0        0 eno1
~ # ping 10.192.0.1
PING 10.192.0.1 (10.192.0.1): 56 data bytes
^C
--- 10.192.0.1 ping statistics ---
6 packets transmitted, 0 packets received, 100% packet loss
Marostegui triaged this task as High priority.Jan 14 2020, 6:51 AM

So, I digged into this a little: Interface auto setup not working if one the devices has no link seems to be a known bug: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=700292

If we don't have a clean way to disable the 10G card until it's actually linked/used, we can try passing the MAC address of the 1G interface via BOOTIF=aa-bb-cc-dd-ee-ff in the pxelinux.cfg (https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=615600)

@Papaul can you double check (maybe even with the vendor) if there is a way to disable the 10G port for now?

So, I digged into this a little: Interface auto setup not working if one the devices has no link seems to be a known bug: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=700292

If we don't have a clean way to disable the 10G card until it's actually linked/used, we can try passing the MAC address of the 1G interface via BOOTIF=aa-bb-cc-dd-ee-ff in the pxelinux.cfg (https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=615600)

BOOTIF doesn't seem to be making any difference, the host keeps failing.
While the option shows on the cmdline with the correct mac:

BOOT_IMAGE=debian-installer/amd64/linux initrd=debian-installer/amd64/initrd.gz vga=normal auto-install/enable=true preseed/url=http://apt.wikimedia.org/autoinstall/preseed.cfg DEBCONF_DEBUG=5 netcfg/choose_interface=auto netcfg/get_hostname=unassigned netcfg/get_domain=unassigned netcfg/dhcp_timeout=60 BOOTIF=b0-26-28-f5-28-02 --- console=ttyS1,115200n8

Which is eno3:

4: eno3: <BROADCAST,MULTICAST> mtu 1500 qdisc mq qlen 1000
    link/ether b0:26:28:f5:28:02 brd ff:ff:ff:ff:ff:ff
~ #    ip addr add 10.192.0.157/22 dev eno3
~ # ip link set eno3 up
~ #  ping 10.192.0.1
PING 10.192.0.1 (10.192.0.1): 56 data bytes
64 bytes from 10.192.0.1: seq=0 ttl=64 time=1006.032 ms
64 bytes from 10.192.0.1: seq=1 ttl=64 time=5.930 ms
64 bytes from 10.192.0.1: seq=2 ttl=64 time=0.393 ms
64 bytes from 10.192.0.1: seq=3 ttl=64 time=0.428 ms

When I attempt to configure the network manually for the installing it keeps choosing eno1, which is the 10G without link.
I have found some bugs regarding BOOTIF, but doesn't seem to be affecting us, ie: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=826844

So adding the 01- did the trick and es2020 is installing:

append initrd=debian-installer/amd64/initrd.gz vga=normal auto-install/enable=true preseed/url=http://apt.wikimedia.org/autoinstall/preseed.cfg DEBCONF_DEBUG=5 netcfg/choose_interface=auto netcfg/get_hostname=unassigned netcfg/get_domain=unassigned netcfg/dhcp_timeout=60 BOOTIF=01-b0-26-28-f5-28-02 --- console=ttyS1,115200n8

This is a workaround, but we have to install 12 hosts in total (6 per DC), @Papaul @wiki_willy can we double check with Dell if (and how) the 10G can be disabled on BIOS or somewhere else?

Forgot to thank @MoritzMuehlenhoff for all the help and time with the troubleshooting

On the phone with Dell support.

Dell said that it is not possible to disable the 10Gb interface.

  • Enable to 10G even though it will go to a 1G switch port? Is that even possible?

Not afaik.

  • Enable to 10G even though it will go to a 1G switch port? Is that even possible?

Not afaik.

Is that because of different cables/connectors?

Is that because of different cables/connectors?

Indeed, 1G switch ports are RJ45, 10G are SFP. We could try to put an SFP-T on the server side. But I don't think it will work.

So I asked Dell if it was possible to replace the NIC card we have now with 2 separate NiC cards ( 1 x10Gb NiC and 1 x1GB NIC). He is checking with our account rep to see if it is possible and he will get back in touch with me.

Change 564724 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] install_server: Changing es2023 MAC to the 10G one

https://gerrit.wikimedia.org/r/564724

Change 564724 merged by Marostegui:
[operations/puppet@production] install_server: Changing es2023 MAC to the 10G one

https://gerrit.wikimedia.org/r/564724

pxelinux has a generic option to pass the MAC in place when receiving the boot image as BOOTIF: https://wiki.syslinux.org/wiki/index.php?title=SYSLINUX#SYSAPPEND_bitmask

Let's try a reimage os es2020 with

ipappend 2

added to /srv/tftpboot/stretch-installer/pxelinux.cfg/ttyS1-115200 (between the lines starting with "kernel" and "append initrd" to confirm.

Mentioned in SAL (#wikimedia-operations) [2020-01-14T16:26:17Z] <marostegui> Disable temporarily puppet on install1002 and install2002 - T242481

Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts:

es2020.codfw.wmnet

The log can be found in /var/log/wmf-auto-reimage/202001141627_marostegui_202850_es2020_codfw_wmnet.log.

pxelinux has a generic option to pass the MAC in place when receiving the boot image as BOOTIF: https://wiki.syslinux.org/wiki/index.php?title=SYSLINUX#SYSAPPEND_bitmask

Let's try a reimage os es2020 with

ipappend 2

added to /srv/tftpboot/stretch-installer/pxelinux.cfg/ttyS1-115200 (between the lines starting with "kernel" and "append initrd" to confirm.

This worked!

Mentioned in SAL (#wikimedia-operations) [2020-01-14T16:41:24Z] <marostegui> Enable puppet back on install1002 and install2002 - T242481

Change 564729 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Pass down MAC address of to installing system via BOOTIF

https://gerrit.wikimedia.org/r/564729

Completed auto-reimage of hosts:

['es2020.codfw.wmnet']

and were ALL successful.

MoritzMuehlenhoff renamed this task from Missing Network drivers from Stretch and Buster installer for BRCM 2P 1G BT + 2P 10G SFP NDC to d-i fails to install on servers with BRCM 2P 1G BT + 2P 10G SFP NDC.Jan 15 2020, 11:17 AM
MoritzMuehlenhoff updated the task description. (Show Details)

This install error is caused by the fact that these servers have a dual port NIC with 1G and 10G interfaces, but only the 1G interface has a link currently (as we miss a 10G switch in that rack). This causes the error from this task, it's a long-standing bug in netcfg: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=700292

There's however a workaround: If the MAC address of the NIC from which d-i was PXE-booted is passed in the form BOOTIF=01-$MAC, then netcfg uses this MAC to determine the NIC. This was added in netcfg 1.63 (https://tracker.debian.org/news/504452/accepted-netcfg-163-source-i386/)

pxelinux has a generic option ("ipappend 2") to pass the MAC in place when receiving the boot image as BOOTIF down to d-i: https://wiki.syslinux.org/wiki/index.php?title=SYSLINUX#SYSAPPEND_bitmask

Valentín ran into a different issue last year which required BOOTIF and submitted a patch to d-i to prevent BOOTIF being passed to the installed system: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=921444
This change landed in Buster, but it's not available in Stretch.

It is however currently needed to install DB hosts which ATM need Stretch still. I did a codesearch for "BOOTIF" and there's a number of packages which parse/process BOOTIF. The presense of BOOTIF on the installed system is probably just cosmestic, but it's not easy to rule it out conclusively.

I see a number of options:

  1. Submit a patch to backport the change to the Stretch 9.12 point release scheduled for February 8th. Probably too late for the current es2* installs, though and we can't simply rely that this gets backported.
  2. Take the risk and enable for stretch/buster
  3. Enable "ipappend 2" only for Buster and create a separate tftpboot environment /srv/tftpboot/stretch-installer-bootif which enables "ipappend 2". The es2* hosts can then simply get that assigned in the DHCP config and the impact of that change is strictly contained to the es2* hosts. Plus, if we also do 1. in parallel, we can simply revert to the standard stretch config if it gets merged in 9.12 (and these are going to be reimaged to Buster in this year anyway)

I'm inclined to go with 3.

I like option 3 indeed.
There is one more option which is a bit more painful, which is to live hack pxelinux and set the MAC address of the host we are going to install and do it manually. We have 6 hosts in eqiad and 12 in codfw.

Change 565237 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Install es2020 with stretch-installer-bootif TFTP environment

https://gerrit.wikimedia.org/r/565237

Change 565237 merged by Muehlenhoff:
[operations/puppet@production] Install es2020 with stretch-installer-bootif TFTP environment

https://gerrit.wikimedia.org/r/565237

Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts:

['es2020.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202001161014_marostegui_138642.log.

Completed auto-reimage of hosts:

['es2020.codfw.wmnet']

Of which those FAILED:

['es2020.codfw.wmnet']

Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts:

['es2020.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202001161221_marostegui_159400.log.

Completed auto-reimage of hosts:

['es2020.codfw.wmnet']

Of which those FAILED:

['es2020.codfw.wmnet']

Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts:

['es2020.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202001161221_marostegui_159494.log.

Change 565265 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] install_server: Add stretch-installer-bootif to es20XX hosts

https://gerrit.wikimedia.org/r/565265

Change 565265 merged by Marostegui:
[operations/puppet@production] install_server: Add stretch-installer-bootif to es20XX hosts

https://gerrit.wikimedia.org/r/565265

Completed auto-reimage of hosts:

['es2020.codfw.wmnet']

and were ALL successful.

So, this will be addressed in two parts:

Update from Dell after today's meeting.

They are looking for a way to see if it is possible and doable to rename the interfaces on the card so that the 1GB interface can show up as 1-1-1 and not 1-3-1.
They will get back in touch with me.

Last update from Dell below. Nothing we don't know about already.

I have not found the ability to disable a single port on the card. Only the whole card:

cid:image003.png@01D5CC83.D3EC79D0

There is no setting to disable single ports on the NDC (Network daughter card).

I will ask a few others to see if they have any additional ideas.

Thanks Papaul! I think we don't need to pursue the "let's disable the unused port" option further, the current solution within the debian-installer addresses this just fine (plus if we disable the 10G port, it'll cause further effort down the road to re-enable this once we have a 10G rack).

Papaul closed this task as Resolved.Jan 17 2020, 2:20 PM
Papaul claimed this task.

Thansk @MoritzMuehlenhoff. resolving this task

Change 564729 merged by Muehlenhoff:
[operations/puppet@production] Pass down MAC address of the installing system via BOOTIF on Buster

https://gerrit.wikimedia.org/r/564729

Change 573069 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] install_server: Pass bootif installer to new ES hosts

https://gerrit.wikimedia.org/r/573069

Change 573069 merged by Marostegui:
[operations/puppet@production] install_server: Pass bootif installer to new ES hosts

https://gerrit.wikimedia.org/r/573069