We have order some new es servers with BRCM 2P 1G BT + 2P 10G SFP NDC Nic's and when doing the installation both on Stretch and Buster it fails with the error below.
───┤ [!!] Configure the network ├─────────────────────┐ │ Network autoconfiguration failed │ │ Your network is probably not using the DHCP protocol. Alternatively, │ │ the DHCP server may be slow or some network hardware is not working │ │ properly. │ │ │ │ <Continue>
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | Marostegui | T226704 Setup es4 and es5 replica sets for new read-write external store service | |||
| Unknown Object (Task) | |||||
| Resolved | Marostegui | T243052 Productionize es1020-es1025, es2020-es2025 | |||
| Resolved | Papaul | T241336 (Needed By 31st January) codfw: rack/setup/install es202[0-5].codfw.wmnet | |||
| Resolved | Papaul | T242481 d-i fails to install on servers with BRCM 2P 1G BT + 2P 10G SFP NDC |
What if you try to configure the network manually rather than using DHCP, does it fail too?
If you send me the MAC addresses, I can try to look for them on the DHCP servers to see if I can at least see the request or if that even doesn't arrive.
When reseting es2021 via IDRAC I saw this and after a couple of powercycles the host booted up:
!!!! X64 Exception Type - 0E(#PF - Page-Fault) CPU Apic ID - 00000000 !!!! ExceptionData - 0000000000000000 I:0 R:0 U:0 W:0 P:0 PK:0 S:0 RIP - 0000000048167B33, CS - 0000000000000038, RFLAGS - 0000000000010012 RAX - 0000000000000000, RCX - 000000005DA4EFD0, RDX - 000000005D91A028 RBX - 00000000481482C0, RSP - 0000000048147880, RBP - 000000002BBFF8CC RSI - 0000000000000000, RDI - 000000004B6468D8 R8 - 0000000000000013, R9 - 0000000000000000, R10 - 00000000000000C0 R11 - 000000005D91A020, R12 - 0000000000000000, R13 - 0000000000000000 R14 - 0000000000000000, R15 - 0000000000000000 DS - 0000000000000018, ES - 0000000000000018, FS - 0000000000000018 GS - 0000000000000018, SS - 0000000000000018 CR0 - 0000000080010013, CR2 - FFFFFFFFE8FD1414, CR3 - 0000000047E01000 CR4 - 0000000000000628, CR8 - 0000000000000000 DR0 - 0000000000000000, DR1 - 0000000000000000, DR2 - 0000000000000000 DR3 - 0000000000000000, DR6 - 00000000FFFF0FF0, DR7 - 0000000000000400 GDTR - 000000004B647D78 0000000000000047, LDTR - 0000000000000000 IDTR - 00000000475FD008 00000000000001FF, TR - 0000000000000000 FXSAVE_STATE - 00000000481474E0 !!!! Find image based on IP(0x48167B33) u:\Build\DellAtlasPkg\DEBUG_MYTOOLS\X64\DellPkgs\DellCorePkgs\DellRestrictedModulePkg\Override\MdeModulePkg\Core\Dxe\DxeMain\DEBUG\DxeCore.pdb (ImageBase=0000000048148000, EntryPoint=00000000481482C0) !!!!
Once booting it logged this:
0 Non-RAID Disk(s) found on the host adapter 0rNon-RAIDeDisk(s) handledtbyrBIOSBoot Agent PowerEdge Expandable0RAIDrControlleriBIOSion 1AVirtualsDrive(s)AfoundRonHthePhostpadapter. 1WVirtualeDrive(s).handledfbyuBIOSonMUtility PowerEdge R740xd BIOS Version: 2.3.10 Console Redirection Enabled Requested by iDRAC Attempting PXE Boot iDRAC IPV4: 10.193.1.72 Initializing Firmware Interfaces... PowerEdge R740xd - BIOS 2.3.10 A system restart is required. The system detected an exception during the UEFI pre-boot environment. ------------------------------------------------------------------------------- Type: Machine Check Error (18) Source: Hardware on BSP AX=00000000581F873C BX=0000000000000000 SI=0000000000000000 DI=0000000000000000 CX=00000000460843BF DX=0000000045EFE284 R8=00000000001C05B6 R9=00000000001C3B9B 10=00000000000000C0 11=0000000048146160 12=0000000000000000 13=0000000046233AA8 14=00000000588F40D8 15=0000000046233AA0 BP=0000000048146CA8 SP=00000000481461A0 IP=00000000460F4BC0 Flags=00010206 CurrentTPL = 04, LastEventTime FFFFFFFFE812900A LastMsg: LBRfr22 460F4BC5 Unknown(ttwgg) LBRto22 460F4B88 Unknown(ttwgg) LBRfr21 460F4BC5 Unknown(ttwgg) LBRto21 460F4B88 Unknown(ttwgg) LBRfr20 460F4BC5 Unknown(ttwgg) LBRto20 460F4B88 Unknown(ttwgg) LBRfr19 460F4BC5 Unknown(ttwgg) LBRto19 460F4B88 Unknown(ttwgg) LBRfr18 460F4BC5 Unknown(ttwgg) LBRto18 460F4B88 Unknown(ttwgg) LBRfr17 460F4BC5 Unknown(ttwgg) LBRto17 460F4B88 Unknown(ttwgg) LBRfr16 460F4BC5 Unknown(ttwgg) LBRto16 460F4B88 Unknown(ttwgg) LBRfr15 460F4BC5 Unknown(ttwgg) LBRto15 460F4B88 Unknown(ttwgg) LBRfr14 460F4BC5 Unknown(ttwgg) LBRto14 460F4B88 Unknown(ttwgg) LBRfr13 460F4BC5 Unknown(ttwgg) LBRto13 460F4B88 Unknown(ttwgg) LBRfr12 460F4BC5 Unknown(ttwgg) LBRto12 460F4B88 Unknown(ttwgg) LBRfr11 460F4BC5 Unknown(ttwgg) LBRto11 460F4B88 Unknown(ttwgg) LBRfr10 460F4BC5 Unknown(ttwgg) LBRto10 460F4B88 Unknown(ttwgg) LBRfr9 460F4BC5 Unknown(ttwgg) LBRto9 460F4B88 Unknown(ttwgg) LBRfr8 460F4BC5 Unknown(ttwgg) LBRto8 460F4B88 Unknown(ttwgg) LBRfr7 460F4BC5 Unknown(ttwgg) LBRto7 460F4B88 Unknown(ttwgg) LBRfr6 460F4BC5 Unknown(ttwgg) LBRto6 460F4B88 Unknown(ttwgg) LBRfr5 460F4BC5 Unknown(ttwgg) LBRto5 460F4B88 Unknown(ttwgg) LBRfr4 460F4BC5 Unknown(ttwgg) LBRto4 460F4B88 Unknown(ttwgg) LBRfr3 460F4BC5 Unknown(ttwgg) LBRto3 460F4B88 Unknown(ttwgg) LBRfr2 460F4BC5 Unknown(ttwgg) LBRto2 460F4B88 Unknown(ttwgg) LBRfr1 460F4BC5 Unknown(ttwgg) LBRto1 460F4B88 Unknown(ttwgg) LBRfr0 460F4BC5 Unknown(ttwgg) LBRto0 460F4B88 Unknown(ttwgg) -->RIP 460F4BC0 Unknown(ttwgg) s00 460F4872 Unknown(ttwgg) s01 460F7E1C Unknown(ttwgg) s02 460FC36E Unknown(ttwgg) s03 460FA8A8 Unknown(ttwgg) s04 462EA25D Unknown(ngkkh) +01025D Avago EFI SAS Driver s05 462EA3B0 Unknown(ngkkh) +0103B0 Avago EFI SAS Driver s06 481698AB DxeCore.efi +0218AB s07 48168BE7 DxeCore.efi +020BE7 s08 481606A2 DxeCore.efi +0186A2 s09 48168F97 DxeCore.efi +020F97 s10 481606A2 DxeCore.efi +0186A2 s11 46DBB033 DellBdsDxe.efi +01A033 s12 46DBB0A3 DellBdsDxe.efi +01A0A3 s13 46DBB19D DellBdsDxe.efi +01A19D s14 46DA447E DellBdsDxe.efi +00347E s15 46DA589A DellBdsDxe.efi +00489A s16 48149F5B DxeCore.efi +001F5B s17 48148873 DxeCore.efi +000873 s18 481482DF DxeCore.efi +0002DF s19 4B646004 Unknown(hfdzw) +006004 Stack Dump: 481461A0 00000000581F873C <..X.... 481461A8 00000000460843BF .C.F.... 481461B0 0000000000000958 X....... 481461B8 00000000460F4872 rH.F.... 481461C0 0000000058072601 .&.X.... 481461C8 0000000045EFE284 ...E.... 481461D0 000000000018613B ;a...... 481461D8 000000004610503B ;P.F.... 481461E0 0000000000000004 ........ 481461E8 00000000460F7E1C .~.F.... 481461F0 0000000058072601 .&.X.... 481461F8 0000000045EFE284 ...E.... 48146200 00000000001C05B6 ........ 48146208 0000000000000000 ........ 48146210 000000005806F020 ..X.... 48146218 00000000001C05B6 ........ 48146220 0000000048146280 .b.H.... 48146228 0000000000000000 ........ 48146230 0000000045EFE280 ...E.... 48146238 FFFFFFFFFFFFFFFF ........ 48146240 0000000058072601 .&.X.... 48146248 0000000000000000 ........ 48146250 0000000000000000 ........ 48146258 00000000460FC36E n..F.... 48146260 00000000460C1C90 ...F.... 48146268 00000000588FEC20 ..X.... 48146270 00000000460DE6A0 ...F.... 48146278 0000000045EFE280 ...E.... 48146280 0000000000000000 ........ 48146288 0000000000000958 X....... 48146290 0000000000000010 ........ 48146298 000000004816E14D M..H.... 481462A0 0000000000000000 ........ 481462A8 0000000000000000 ........ 481462B0 0100000100000102 ........ 481462B8 C128710000000100 .....q(. 481462C0 C128737CC02DA078 x.-.|s(. 481462C8 0000000048154360 `C.H.... 481462D0 1F421028005D1000 ..].(.B. 481462D8 0000000030430000 ..C0.... 481462E0 0000000000000000 ........ 481462E8 0000000000000000 ........ 481462F0 0000000000030302 ........ 481462F8 0000000000000000 ........ 48146300 0000000000000000 ........ 48146308 0000000000000000 ........ 48146310 0000000000000000 ........ 48146318 0000000000000000 ........ 48146320 0000000000000000 ........ 48146328 0000000000000000 ........ 48146330 0000000000000000 ........ 48146338 0800000000000020 ....... 48146340 500056B374B516FF ...t.V.P 48146348 0000000000000000 ........ 48146350 0000000000000000 ........ 48146358 0000000000000000 ........ 48146360 0000000000000000 ........ 48146368 0000000000000000 ........ 48146370 0000000000000000 ........ 48146378 0000000000000000 ........ 48146380 0000000838303133 3108.... 48146388 00000000534F4942 BIOS.... 48146390 2E31302E33332E36 6.33.01. 48146398 302E36312E345F30 0_4.16.0 481463A0 3078305F30302E37 7.00_0x0 481463A8 0034303330323136 6120304. 481463B0 30322F32322F3230 02/22/20 481463B8 0000000000003931 19...... 481463C0 30322F32322F3230 02/22/20 481463C8 0000000000003931 19...... 481463D0 00000000524C5443 CTLR.... 481463D8 3037302D38312E35 5.18-070 481463E0 0000000000000032 2....... 481463E8 0000000000000000 ........ 481463F0 0000000000000000 ........ 481463F8 3220393220677541 Aug 29 2 48146400 0000000000383130 018..... 48146408 37333A35343A3631 16:45:37 48146410 0000000000000000 ........ 48146418 00000000444C5043 CPLD.... 48146420 0000000030323030 0020.... 48146428 0000000000000000 ........ 48146430 0000000000000000 ........ 48146438 0000000000000000 ........ 48146440 37312F36322F3031 10/26/17 48146448 0000000000000000 ........ 48146450 33333A30353A3731 17:50:33 48146458 0000000000000000 ........ 48146460 0000000020505041 APP .... 48146468 30302E3030332E34 4.300.00 48146470 000000323533382D -8352... 48146478 0000000000000000 ........ 48146480 0000000000000000 ........ 48146488 32203230206C754A Jul 02 2 48146490 0000000000393130 019..... 48146498 38313A30353A3531 15:50:18 481464A0 0000000000000000 ........ 481464A8 000000005444564E NVDT.... 481464B0 302E313135312E33 3.1511.0 481464B8 0000383230302D30 0-0028.. 481464C0 0000000000000000 ........ 481464C8 0000000000000000 ........ 481464D0 322031332074634F Oct 31 2 481464D8 0000000000373130 017..... 481464E0 30303A30313A3731 17:10:00 481464E8 0000000000000000 ........ 481464F0 00000000554D4244 DBMU.... 481464F8 0000000030363030 0060.... 48146500 0000000000000000 ........ 48146508 0000000000000000 ........ 48146510 0000000000000000 ........ 48146518 37312F31332F3830 08/31/17 48146520 0000000000000000 ........ 48146528 30343A33303A3331 13:03:40 48146530 0000000000000000 ........ 48146538 0000000057464753 SGFW.... 48146540 0032302E30302E31 1.00.02. 48146548 0000000000000000 ........ 48146550 0000000000000000 ........ 48146558 0000000000000000 ........ 48146560 3220343220677541 Aug 24 2 48146568 0000000000333130 013..... 48146570 38313A36303A3530 05:06:18 48146578 0000000000000000 ........ 48146580 000000004C425442 BTBL.... 48146588 2E30302E37302E33 3.07.00. 48146590 00333030302D3030 00-0003. 48146598 0000000000000000 ........ 481465A0 0000000000000000 ........ 481465A8 32203133206C754A Jul 31 2 481465B0 0000000000353130 015..... 481465B8 38313A37343A3431 14:47:18 481465C0 0000000000000000 ........ 481465C8 0000000000000000 ........ 481465D0 0000000000000000 ........ 481465D8 0000000000000000 ........ 481465E0 0000000000000000 ........ 481465E8 0000000000000000 ........ 481465F0 0000000000000000 ........ 481465F8 0000000000000000 ........ 48146600 0000000000000000 ........ 48146608 0000000000000000 ........ 48146610 0000000000000000 ........ 48146618 0000000000000000 ........ 48146620 0000000000000000 ........ 48146628 0000000000000000 ........ 48146630 0000000000000000 ........ 48146638 0000000000000000 ........ 48146640 0000000000000000 ........ 48146648 0000000000000000 ........ 48146650 0000000000000000 ........ 48146658 0000000000000000 ........ 48146660 0000000000000000 ........ 48146668 0000000000000000 ........ 48146670 0000000000000000 ........ 48146678 0000000000000000 ........ 48146680 0000000000000000 ........ 48146688 0000000000000000 ........ 48146690 0000000000000000 ........ 48146698 0000000000000000 ........ 481466A0 0000000000000000 ........ 481466A8 0000000000000000 ........ 481466B0 0000000000000000 ........ 481466B8 0000000000000000 ........ 481466C0 0000000000000000 ........ 481466C8 0000000000000000 ........ 481466D0 0000000000000000 ........ 481466D8 0000000000000000 ........ 481466E0 0000000000000000 ........ 481466E8 0000000000000000 ........ 481466F0 0000000000000000 ........ 481466F8 0000000000000000 ........ 48146700 0000000000000000 ........ 48146708 0000000000000000 ........ 48146710 0000000000000000 ........ 48146718 0000000000000000 ........ 48146720 0000000000000000 ........ 48146728 0000000000000000 ........ 48146730 0000000000000000 ........ 48146738 0000000000000000 ........ 48146740 0000000000000000 ........ 48146748 0000000000000000 ........ 48146750 0000000000000000 ........ 48146758 0000000000000000 ........ 48146760 0000000000000000 ........ 48146768 0000000000000000 ........ 48146770 0000000000000000 ........ 48146778 0000000000000000 ........ 48146780 0000000000000000 ........ 48146788 0000000000000000 ........ 48146790 0000000000000000 ........ 48146798 0000000000000000 ........ 481467A0 0000000000000000 ........ 481467A8 0000000000000000 ........ 481467B0 0000000000000000 ........ 481467B8 0000000000000000 ........ 481467C0 0000000000000000 ........ 481467C8 0000000000000000 ........ 481467D0 0000000000000000 ........ 481467D8 0000000000000000 ........ 481467E0 0000000000000000 ........ 481467E8 0000000000000000 ........ 481467F0 0000000000000000 ........ 481467F8 0000000000000000 ........ 48146800 0000000000000000 ........ 48146808 4080082000000000 .... ..@ 48146810 3337482043524550 PERC H73 48146818 7470616441205030 0P Adapt 48146820 0000000000007265 er...... 48146828 0000000000000000 ........ 48146830 0000000000000000 ........ 48146838 0000000000000000 ........ 48146840 0000000000000000 ........ 48146848 0000000000000000 ........ 48146850 0000000000000000 ........ 48146858 0000000000000000 ........ 48146860 00504F3430314239 9B104OP. 48146868 0000000000000000 ........ 48146870 0000000000000000 ........ 48146878 0000000000000000 ........ 48146880 25ABDB2500000C3D =...%..% 48146888 00002000003C03A0 ..<.. .. 48146890 000D000000000001 ........ 48146898 002000000000000C ...... . 481468A0 0000000000100800 ........ 481468A8 00000EF700800000 ........ 481468B0 00005E3F1DF45FBF ._..?^.. 481468B8 00042D0700000B07 .....-.. 481468C0 0000000000000007 ........ 481468C8 0000000000030000 ........ 481468D0 00320010012C0006 ..,...2. 481468D8 0C04041E1E1E1E1E ........ 481468E0 00F0FF0000000100 ........ 481468E8 0000000000000000 ........ 481468F0 001E00000402704C Lp...... 481468F8 00000806000000FE ........ 48146900 000000000E040000 ........ 48146908 00000000000A012C ,....... 48146910 302E362E352E3532 25.5.6.0 48146918 0000000000393030 009..... 48146920 0000000000000000 ........ 48146928 0000000000000000 ........ 48146930 0000000000000000 ........ 48146938 0000000000000000 ........ 48146940 0000000000000000 ........ 48146948 0000000000000000 ........ 48146950 0000000000000000 ........ 48146958 0000000000000000 ........ 48146960 0000000000000000 ........ 48146968 0000000000000000 ........ 48146970 0000000000000000 ........ 48146978 0000000000000000 ........ 48146980 0000000000000000 ........ 48146988 0000000000000000 ........ 48146990 0000000000000000 ........ 48146998 0000000000000000 ........ 481469A0 0000000000000000 ........ 481469A8 0000000000000000 ........ 481469B0 0000000000000000 ........ 481469B8 0000000000000000 ........ 481469C0 0000000000000000 ........ 481469C8 0000000000000000 ........ 481469D0 0000000000000000 ........ 481469D8 0000000000000000 ........ 481469E0 0000000000000000 ........ 481469E8 0000000000000000 ........ 481469F0 0000000000000000 ........ 481469F8 0000000000000000 ........ 48146A00 0000000000000000 ........ 48146A08 0000000000000000 ........ 48146A10 0000000000000000 ........ 48146A18 0000000000000000 ........ 48146A20 0000000000000000 ........ 48146A28 0000000000000000 ........ 48146A30 0203020300220201 .."..... 48146A38 1006100601040203 ........ 48146A40 0000000010080204 ........ 48146A48 0000000000000000 ........ 48146A50 0200010000FE0100 ........ 48146A58 0020000000001010 ...... . 48146A60 0000000000000000 ........ 48146A68 0000000000000000 ........ 48146A70 40176AA4078B0000 .....j.@ 48146A78 0000000000000000 ........ 48146A80 0000000000000000 ........ 48146A88 0000000000000000 ........ 48146A90 0000000000000000 ........ 48146A98 000000F8003C3C00 .<<..... 48146AA0 0000000000000000 ........ 48146AA8 0000000000000000 ........ 48146AB0 0000000000000000 ........ 48146AB8 0000000042520825 %.RB.... 48146AC0 0000000000000000 ........ 48146AC8 0000000000000000 ........ 48146AD0 0000000000000000 ........ 48146AD8 0000000000000000 ........ 48146AE0 0000000000000000 ........ 48146AE8 0000000000000000 ........ 48146AF0 0000000000000000 ........ 48146AF8 0000000000000000 ........ 48146B00 0000000000000000 ........ 48146B08 0000000000000000 ........ 48146B10 0000000000000000 ........ 48146B18 0000000000000000 ........ 48146B20 0000000000000000 ........ 48146B28 0000000000000000 ........ 48146B30 0000000000000000 ........ 48146B38 0000000000000000 ........ 48146B40 0000000000000000 ........ 48146B48 0000000000000000 ........ 48146B50 0000000000000000 ........ 48146B58 0000000000000000 ........ 48146B60 0000000000000000 ........ 48146B68 0000000000000000 ........ 48146B70 0000000000000000 ........ 48146B78 0000000000000000 ........ 48146B80 0000000000000000 ........ 48146B88 0000000000000000 ........ 48146B90 0000000000000000 ........ 48146B98 0000000000000000 ........ 48146BA0 0000000000000000 ........ 48146BA8 0000000000000000 ........ 48146BB0 0000000000000000 ........ 48146BB8 0000000000000000 ........ 48146BC0 0000000000000000 ........ 48146BC8 0000000000000000 ........ 48146BD0 0000000000000000 ........ 48146BD8 0000000000000000 ........ 48146BE0 0000000000000000 ........ 48146BE8 0000000000000000 ........ 48146BF0 0000000000000000 ........ 48146BF8 0000000000000000 ........ 48146C00 0000000000000000 ........ 48146C08 0000000000000000 ........ 48146C10 0000000000000000 ........ 48146C18 0000000000000000 ........ 48146C20 54CD98F0AAE52300 .#.....T 48146C28 0000020104071E00 ........ 48146C30 0000010000010100 ........ 48146C38 0000000000000001 ........ 48146C40 0000000000000000 ........ 48146C48 00000000582BA520 .+X.... 48146C50 0000000000000000 ........ 48146C58 00000000582BA700 ..+X.... 48146C60 00000000582B3020 0+X.... 48146C68 00000000460FA8A8 ...F.... 48146C70 00000000582BA520 .+X.... 48146C78 00000000460E3A68 h:.F.... 48146C80 8000000000000002 ........ 48146C88 0000000000000000 ........ 48146C90 0000000000000000 ........ 48146C98 00000000462EA25D ]..F.... 48146CA0 00000000588FEC02 ...X.... 48146CA8 00000000588FEC20 ..X.... 48146CB0 0000000048147728 (w.H.... 48146CB8 00000000588FF220 ..X.... 48146CC0 00000000582B6720 g+X.... 48146CC8 0000000000000008 ........ 48146CD0 00000000582B3020 0+X.... 48146CD8 0000000000000000 ........ 48146CE0 0000000000000000 ........ 48146CE8 0000000000000000 ........ 48146CF0 00000000582BA700 ..+X.... 48146CF8 000000004BAD37FE .7.K.... 48146D00 3FFFFFFF00000002 .......? 48146D08 0000000000000001 ........ 48146D10 0000000001020201 ........ 48146D18 0000000000000000 ........ 48146D20 54CD98F0AAE52300 .#.....T 48146D28 0000020104071E00 ........ 48146D30 0000010000010100 ........ 48146D38 0100000004000001 ........ 48146D40 0001010000000001 ........ 48146D48 0000000000010001 ........ 48146D50 7F00000000010101 ........ 48146D58 015001000000001E ......P. 48146D60 0000013C00040075 u...<... 48146D68 0000000000000000 ........ 48146D70 1F421028005D1000 ..].(.B. 48146D78 0000000030430000 ..C0.... 48146D80 0000000000000000 ........ 48146D88 0000000000000000 ........ 48146D90 0000000000030302 ........ 48146D98 0000000000000000 ........ 48146DA0 0000000000000000 ........ 48146DA8 0000000000000000 ........ 48146DB0 0000000000000000 ........ 48146DB8 0000000000000000 ........ 48146DC0 0000000000000000 ........ 48146DC8 0000000000000000 ........ 48146DD0 0000000000000000 ........ 48146DD8 0800000000000020 ....... 48146DE0 500056B374B516FF ...t.V.P 48146DE8 0000000000000000 ........ 48146DF0 0000000000000000 ........ 48146DF8 0000000000000000 ........ 48146E00 0000000000000000 ........ 48146E08 0000000000000000 ........ 48146E10 0000000000000000 ........ 48146E18 0000000000000000 ........ 48146E20 0000000838303133 3108.... 48146E28 00000000534F4942 BIOS.... 48146E30 2E31302E33332E36 6.33.01. 48146E38 302E36312E345F30 0_4.16.0 48146E40 3078305F30302E37 7.00_0x0 48146E48 0034303330323136 6120304. 48146E50 30322F32322F3230 02/22/20 48146E58 0000000000003931 19...... 48146E60 30322F32322F3230 02/22/20 48146E68 0000000000003931 19...... 48146E70 00000000524C5443 CTLR.... 48146E78 3037302D38312E35 5.18-070 48146E80 0000000000000032 2....... 48146E88 0000000000000000 ........ 48146E90 0000000000000000 ........ 48146E98 3220393220677541 Aug 29 2 48146EA0 0000000000383130 018..... 48146EA8 37333A35343A3631 16:45:37 48146EB0 0000000000000000 ........ 48146EB8 00000000444C5043 CPLD.... 48146EC0 0000000030323030 0020.... 48146EC8 0000000000000000 ........ 48146ED0 0000000000000000 ........ 48146ED8 0000000000000000 ........ 48146EE0 37312F36322F3031 10/26/17 48146EE8 0000000000000000 ........ 48146EF0 33333A30353A3731 17:50:33 48146EF8 0000000000000000 ........ 48146F00 0000000020505041 APP .... 48146F08 30302E3030332E34 4.300.00 48146F10 000000323533382D -8352... 48146F18 0000000000000000 ........ 48146F20 0000000000000000 ........ 48146F28 32203230206C754A Jul 02 2 48146F30 0000000000393130 019..... 48146F38 38313A30353A3531 15:50:18 48146F40 0000000000000000 ........ 48146F48 000000005444564E NVDT.... 48146F50 302E313135312E33 3.1511.0 48146F58 0000383230302D30 0-0028.. 48146F60 0000000000000000 ........ 48146F68 0000000000000000 ........ 48146F70 322031332074634F Oct 31 2 48146F78 0000000000373130 017..... 48146F80 30303A30313A3731 17:10:00 48146F88 0000000000000000 ........ 48146F90 00000000554D4244 DBMU.... 48146F98 0000000030363030 0060.... 48146FA0 0000000000000000 ........ 48146FA8 0000000000000000 ........ 48146FB0 0000000000000000 ........ 48146FB8 37312F31332F3830 08/31/17 48146FC0 0000000000000000 ........ 48146FC8 30343A33303A3331 13:03:40 48146FD0 0000000000000000 ........ 48146FD8 0000000057464753 SGFW.... 48146FE0 0032302E30302E31 1.00.02. 48146FE8 0000000000000000 ........ 48146FF0 0000000000000000 ........ 48146FF8 0000000000000000 ........ 48147000 3220343220677541 Aug 24 2 48147008 0000000000333130 013..... 48147010 38313A36303A3530 05:06:18 48147018 0000000000000000 ........ 48147020 000000004C425442 BTBL.... 48147028 2E30302E37302E33 3.07.00. 48147030 00333030302D3030 00-0003. 48147038 0000000000000000 ........ 48147040 0000000000000000 ........ 48147048 32203133206C754A Jul 31 2 48147050 0000000000353130 015..... 48147058 38313A37343A3431 14:47:18 48147060 0000000000000000 ........ 48147068 0000000000000000 ........ 48147070 0000000000000000 ........ 48147078 0000000000000000 ........ 48147080 0000000000000000 ........ 48147088 0000000000000000 ........ 48147090 0000000000000000 ........ 48147098 0000000000000000 ........ 481470A0 0000000000000000 ........ 481470A8 0000000000000000 ........ 481470B0 0000000000000000 ........ 481470B8 0000000000000000 ........ 481470C0 0000000000000000 ........ 481470C8 0000000000000000 ........ 481470D0 0000000000000000 ........ 481470D8 0000000000000000 ........ 481470E0 0000000000000000 ........ 481470E8 0000000000000000 ........ 481470F0 0000000000000000 ........ 481470F8 0000000000000000 ........ 48147100 0000000000000000 ........ 48147108 0000000000000000 ........ 48147110 0000000000000000 ........ 48147118 0000000000000000 ........ 48147120 0000000000000000 ........ 48147128 0000000000000000 ........ 48147130 0000000000000000 ........ 48147138 0000000000000000 ........ 48147140 0000000000000000 ........ 48147148 0000000000000000 ........ 48147150 0000000000000000 ........ 48147158 0000000000000000 ........ 48147160 0000000000000000 ........ 48147168 0000000000000000 ........ 48147170 0000000000000000 ........ 48147178 0000000000000000 ........ 48147180 0000000000000000 ........ 48147188 0000000000000000 ........ 48147190 0000000000000000 ........ 48147198 0000000000000000 ........ LBRfr0 Dump: 460F4BA5 FF C8 48 89 44 24 08 48 8B 04 24 48 FF C8 48 89 460F4BB5 04 24 48 8B 04 24 48 8B 4C 24 08 0F B6 09 88 08 460F4BC5 EB C1 48 8B 44 24 20 48 83 C4 18 C3 CC CC CC CC 460F4BD5 CC CC CC CC CC CC CC CC CC CC CC 44 88 44 24 18 RIP Dump: 460F4BA0 8B 44 24 08 48 FF C8 48 89 44 24 08 48 8B 04 24 460F4BB0 48 FF C8 48 89 04 24 48 8B 04 24 48 8B 4C 24 08 460F4BC0 0F B6 09 88 08 EB C1 48 8B 44 24 20 48 83 C4 18 460F4BD0 C3 CC CC CC CC CC CC CC CC CC CC CC CC CC CC CC Machine Check Banks: MC00 STAT=0000000000000000 ADDR=0000000000000000 MISC=0000000000000000 MC01 STAT=BD80000000100134 ADDR=0000000046084380 MISC=0000000000000086 MC02 STAT=0000000000000000 ADDR=0000000000000000 MISC=0000000000000000 MC03 STAT=0000000000000000 ADDR=0000000000000000 MISC=0000000000000000 MC04 STAT=0000000000000000 ADDR=0000000000000000 MISC=0000000000000000 MC05 STAT=0000000000000000 ADDR=0000000000000000 MISC=0000000000000000 MC06 STAT=0000000000000000 ADDR=0000000000000000 MISC=0000000000000000 MC07 STAT=0000000000000000 ADDR=0000000000000000 MISC=0000000000000000 MC08 STAT=0000000000000000 ADDR=0000000046084380 MISC=200400C02FE02086 MC09 STAT=0000000000000000 ADDR=0000000000000000 MISC=0000000000000000 MC10 STAT=0000000000000000 ADDR=0000000000000000 MISC=0000000000000000 MC11 STAT=0000000000000000 ADDR=0000000000000000 MISC=0000000000000000 MC12 STAT=0000000000000000 ADDR=0000000000000000 MISC=0000000000000000 MC13 STAT=0000000000000000 ADDR=0000000000000000 MISC=0000000000000000 MC14 STAT=0000000000000000 ADDR=0000000000000000 MISC=0000000000000000 MC15 STAT=0000000000000000 ADDR=0000000000000000 MISC=0000000000000000 MC16 STAT=0000000000000000 ADDR=000000000AA52BC0 MISC=090821EFB025C086 MC17 STAT=0000000000000000 ADDR=0000000000000000 MISC=0000000000000000 MC18 STAT=0000000000000000 ADDR=0000000000000000 MISC=0000000000000000 MC19 STAT=0000000000000000 ADDR=0000000000000000 MISC=0000000000000000 TSC value when exception occurred: 0095F4E57CF2 CPU TSC frequency : 000095191910 (2,501,450,000) Log of UEFI DxeCore Events: Now-StartTSC Now-EndTSC TPL FuncAddr FFFFFFFFD1551BBE FFFFFFFFD154B27A 1F 468E4B70 XhciDxe.efi +008B70 FFFFFFFFD1551FA6 FFFFFFFFD1551D1A 1F 4816AA20 DxeCore.efi +022A20 FFFFFFFFD17B4D7C FFFFFFFFD17AE3B0 1F 468E4B70 XhciDxe.efi +008B70 FFFFFFFFD17B514E FFFFFFFFD17B4ED0 1F 4816AA20 DxeCore.efi +022A20 FFFFFFFFD1A10836 FFFFFFFFD1A09EB2 1F 468E4B70 XhciDxe.efi +008B70 FFFFFFFFD1A13292 FFFFFFFFD1A13006 1F 4816AA20 DxeCore.efi +022A20 FFFFFFFFD1C72F30 FFFFFFFFD1C6C504 1F 468E4B70 XhciDxe.efi +008B70 FFFFFFFFD1C74284 FFFFFFFFD1C73FE2 1F 4816AA20 DxeCore.efi +022A20 FFFFFFFFD1ED3220 FFFFFFFFD1ECC8B6 1F 468E4B70 XhciDxe.efi +008B70 FFFFFFFFD1ED4D90 FFFFFFFFD1ED4B1A 1F 4816AA20 DxeCore.efi +022A20 FFFFFFFFD21381AC FFFFFFFFD21317D8 1F 468E4B70 XhciDxe.efi +008B70 FFFFFFFFD2138572 FFFFFFFFD2138302 1F 4816AA20 DxeCore.efi +022A20 FFFFFFFFD239244E FFFFFFFFD23921BA 1F 4737CC60 CpuMpDxe.efi +002C60 FFFFFFFFD2398EA6 FFFFFFFFD2392634 1F 468E4B70 XhciDxe.efi +008B70 FFFFFFFFD23993B4 FFFFFFFFD2398FFA 1F 4816AA20 DxeCore.efi +022A20 FFFFFFFFD25F4A8C FFFFFFFFD25EE186 1F 468E4B70 XhciDxe.efi +008B70 FFFFFFFFD25F77D0 FFFFFFFFD25F74C8 1F 4816AA20 DxeCore.efi +022A20 FFFFFFFFD285A800 FFFFFFFFD2853EB0 1F 468E4B70 XhciDxe.efi +008B70 FFFFFFFFD285ABE2 FFFFFFFFD285A954 1F 4816AA20 DxeCore.efi +022A20 FFFFFFFFD2ABB7D4 FFFFFFFFD2AB4F16 1F 468E4B70 XhciDxe.efi +008B70 FFFFFFFFD2ABBB9E FFFFFFFFD2ABB926 1F 4816AA20 DxeCore.efi +022A20 FFFFFFFFD2D179C8 FFFFFFFFD2D110DA 1F 468E4B70 XhciDxe.efi +008B70 FFFFFFFFD2D19F6C FFFFFFFFD2D19CFA 1F 4816AA20 DxeCore.efi +022A20 FFFFFFFFD2F77548 FFFFFFFFD2F70BBC 1F 468E4B70 XhciDxe.efi +008B70 FFFFFFFFD2F7A950 FFFFFFFFD2F7A6CE 1F 4816AA20 DxeCore.efi +022A20 FFFFFFFFD31DDF26 FFFFFFFFD31D761E 1F 468E4B70 XhciDxe.efi +008B70 FFFFFFFFD31DE302 FFFFFFFFD31DE086 1F 4816AA20 DxeCore.efi +022A20 FFFFFFFFD343ECB0 FFFFFFFFD3438354 1F 468E4B70 XhciDxe.efi +008B70 FFFFFFFFD343F0A8 FFFFFFFFD343EE10 1F 4816AA20 DxeCore.efi +022A20 FFFFFFFFD369F888 FFFFFFFFD3698EB0 1F 468E4B70 XhciDxe.efi +008B70 FFFFFFFFD369FC62 FFFFFFFFD369F9E0 1F 4816AA20 DxeCore.efi +022A20 FFFFFFFFD39005D4 FFFFFFFFD38F9C7C 1F 468E4B70 XhciDxe.efi +008B70 FFFFFFFFD39009CA FFFFFFFFD390072C 1F 4816AA20 DxeCore.efi +022A20 FFFFFFFFD3B4B73C FFFFFFFFD3B4B4DE 1F 4737CC60 CpuMpDxe.efi +002C60 FFFFFFFFD3B52198 FFFFFFFFD3B4B8F8 1F 468E4B70 XhciDxe.efi +008B70 FFFFFFFFD3B57282 FFFFFFFFD3B521C0 1F 469F9C70 TerminalDxe.efi +004C70 FFFFFFFFD3B61098 FFFFFFFFD3B572AA 1F 469F9C70 TerminalDxe.efi +004C70 FFFFFFFFD3B61844 FFFFFFFFD3B611EC 1F 4816AA20 DxeCore.efi +022A20 FFFFFFFFD3DBC682 FFFFFFFFD3DB5D54 1F 468E4B70 XhciDxe.efi +008B70 FFFFFFFFD3DBFB28 FFFFFFFFD3DBF81E 1F 4816AA20 DxeCore.efi +022A20 FFFFFFFFD4022EC2 FFFFFFFFD401C62A 1F 468E4B70 XhciDxe.efi +008B70 FFFFFFFFD40232B0 FFFFFFFFD4023018 1F 4816AA20 DxeCore.efi +022A20 FFFFFFFFD427E346 FFFFFFFFD4277A4A 1F 468E4B70 XhciDxe.efi +008B70 FFFFFFFFD4281634 FFFFFFFFD428139A 1F 4816AA20 DxeCore.efi +022A20 FFFFFFFFD44E475A FFFFFFFFD44DDDC8 1F 468E4B70 XhciDxe.efi +008B70 FFFFFFFFD44E4B34 FFFFFFFFD44E48AE 1F 4816AA20 DxeCore.efi +022A20 FFFFFFFFD474562A FFFFFFFFD473ED36 1F 468E4B70 XhciDxe.efi +008B70 FFFFFFFFD47459F6 FFFFFFFFD474577E 1F 4816AA20 DxeCore.efi +022A20 FFFFFFFFD49A60C6 FFFFFFFFD499F728 1F 468E4B70 XhciDxe.efi +008B70 FFFFFFFFD49A64AE FFFFFFFFD49A6226 1F 4816AA20 DxeCore.efi +022A20 FFFFFFFFD4C07066 FFFFFFFFD4C007B2 1F 468E4B70 XhciDxe.efi +008B70 FFFFFFFFD4C07430 FFFFFFFFD4C071BA 1F 4816AA20 DxeCore.efi +022A20 FFFFFFFFD4E6443E FFFFFFFFD4E5DB7E 1F 468E4B70 XhciDxe.efi +008B70 FFFFFFFFD4E657D0 FFFFFFFFD4E65564 1F 4816AA20 DxeCore.efi +022A20 FFFFFFFFD50C8A9A FFFFFFFFD50C212E 1F 468E4B70 XhciDxe.efi +008B70 FFFFFFFFD50C8E86 FFFFFFFFD50C8BE8 1F 4816AA20 DxeCore.efi +022A20 FFFFFFFFD5322C00 FFFFFFFFD53229AE 1F 4737CC60 CpuMpDxe.efi +002C60 FFFFFFFFD53296DC FFFFFFFFD5322DE6 1F 468E4B70 XhciDxe.efi +008B70 FFFFFFFFD5329BEE FFFFFFFFD532982E 1F 4816AA20 DxeCore.efi +022A20 FFFFFFFFD558A474 FFFFFFFFD5583B5A 1F 468E4B70 XhciDxe.efi +008B70 FFFFFFFFD558A8CA FFFFFFFFD558A5C2 1F 4816AA20 DxeCore.efi +022A20 FFFFFFFFD57EB24A FFFFFFFFD57E48D6 1F 468E4B70 XhciDxe.efi +008B70 FFFFFFFFD57EB628 FFFFFFFFD57EB3A6 1F 4816AA20 DxeCore.efi +022A20 FFFFFFFFD5A4BF7E FFFFFFFFD5A456E8 1F 468E4B70 XhciDxe.efi +008B70 Log of UEFI DXE drivers that were started: Now-StartTSC Now-EndTSC FuncAddr 0000E64E6EFC XXXXXXXXXXXX 462EA2DC Unknown(ngkkh) +0102DC Avago EFI SAS Driver 0000E7315D16 0000E72F7EF2 4694FA60 PciBusDxe.efi +000A60 000103A01AD2 0001039E4486 4694FA60 PciBusDxe.efi +000A60 0001081D2A64 0001081B4E3A 4694FA60 PciBusDxe.efi +000A60 0001135DE4A8 0001135BFF82 4694FA60 PciBusDxe.efi +000A60 000121E1CD4A 000121DFADBA 4694FA60 PciBusDxe.efi +000A60 0002847479B6 000142AF1DA0 46350840 cxundi.efi +05F840 0003C7A0FBD8 0002854982E6 463DE840 cxundi.efi +05F840 Log of UEFI DXE drivers that were Stopped: Now-StopTSC Now-EndTSC FuncAddr 002354373E62 00235435E17A 46947CB0 IsaAcpi.efi +000CB0 002354381558 002354377246 4693BFF0 IsaBusDxe.efi +000FF0 0023543C7EA4 00235438156E 46860BF0 IsaSerialDxe.efi +001BF0 002355282192 0023543C7EF0 469F7320 TerminalDxe.efi +002320 00235528F3C6 0023552821CA 46A091A0 ConSplitterDxe.efi +0021A0 0023552A0BCA 00235528F402 46A091A0 ConSplitterDxe.efi +0021A0 0023552AC36C 0023552AC2FE 46A09110 ConSplitterDxe.efi +002110 0023552B99A6 0023552AC3A0 46A09110 ConSplitterDxe.efi +002110 EFI Debug Support Table UpdateStatus: 2 EFI Debug Support Table TableSize : 263 48148000 00068000 DxeCore.efi 4BAF6000 000082C0 PcdDxe.efi 5DDF1000 0000E000 RegAccessDxe.efi 5DDE2000 0000F000 ReportStatusCodeRouterRuntimeDxe.efi 4BAEC000 000090A0 SectionExtractionDxe.efi 4BADA000 00011160 PlatformEarlyDxe.efi 5DDD6000 0000C000 CpuIoDxe.efi 4BAD3000 00006E60 CpuIo2Dxe.efi 4BACB000 00007FE0 DataHubDxe.efi 4BAC4000 000067E0 Legacy8259.efi 4BAB3000 000103E0 CpuArchDxe.efi 4BAA4000 00007680 Metronome.efi 5DDC8000 0000E000 RuntimeDxe.efi 4BA9D000 00006E80 DpcDxe.efi 4BA93000 00009240 RngDxe.efi 4BA79000 00019B20 HstiIhvProviderDxe.efi 4BA71000 00007D00 HttpUtilitiesDxe.efi 4BA61000 0000F040 AcpiTableDxe.efi 4BA4C000 000143A0 DevicePathDxe.efi 4BA3F000 0000CF00 EbcDxe.efi 4BA34000 0000AA00 PchSmbusDxe.efi 4BA2C000 000077C0 LegacyInterrupt.efi 5DDBB000 0000D000 SmmControl.efi 4BA20000 0000B520 PchSerialGpio.efi 4BA18000 00007CE0 WdtDxe.efi 4BA11000 000068E0 IncompatiblePciDeviceSupport.efi 4BA06000 0000A360 SmbiosDxe.efi 4B9FF000 00006E60 SmmCommunicationBufferDxe.efi 4B9F4000 0000A800 S3SaveStateDxe.efi 4B9ED000 00006600 PciHotPlug.efi 5DDAF000 0000C000 NvdimmCommon.efi 5DDA3000 0000C000 NvdimmSmbusCommon.efi 4B9E5000 00007940 Idrac9ChipsetDxe.efi 4B9DE000 00006640 DellIdracInterface.efi 4B9D7000 00006CC0 DellIpmiDebug2.efi 4B9D0000 00006920 DellHiiBrowserCapability.efi 4B9C9000 00006A60 DellPerformanceMeasurementDxe.efi 4B9BC000 0000C040 JsonParserDxe.efi 4B9B4000 00007580 DellCmosAccessDxe.efi 4B9AC000 00007240 DellVamDxe.efi 4B9A3000 000085C0 DellSmaDebug.efi 4B998000 0000A060 DellFruAccessDxe.efi 4B990000 000072E0 IdracPsuBootStatusDxe.efi 4B989000 00006760 DpiNvdimmDxe.efi 4B938000 00050D60 DellRSOD.efi 4B908000 0002F460 HiiDatabase.efi 4B8E1000 00026CC0 DellImageDrawDxe.efi 4B870000 00070E40 Gdriver.efi 4B81A000 00055180 MRE.efi 4B813000 00006FC0 DellNmiButtonDxe.efi 4B80C000 00007000 DpiFlashData2Dxe.efi 4B7F6000 00015DE0 DpiPci2Dxe.efi 4B7EF000 00006F00 DpiCpuDxe.efi 4B7E4000 0000A440 DpiPercDxe.efi 4B7DD000 00006AC0 DpiMemoryDxe.efi 4B7D4000 000083A0 DellSystemPortInfoDxe.efi 4B7CD000 00006400 DpiVrDxe.efi 4B7C2000 0000A5A0 DpiNvmeDxe.efi 4B7BB000 00006840 SmbiosTableListDxe.efi 4B7B4000 00006D80 DellPlatformStaticOverridesDxe.efi 5DD98000 0000B000 DellOemHookStatusCodeHandlerDxe.efi 4B7AA000 00009AE0 HstiPlatformDxe.efi 4B7A3000 00006280 DataHubStdErrDxe.efi 4B79A000 00008F20 GenericMemoryTestDxe.efi 4B792000 00007840 SmartTimer.efi 4B788000 00009BC0 FirmwarePerformanceDxe.efi 4B781000 00006400 DellPercDxe.efi 4B774000 0000C480 PersonalityModuleDataDxe.efi 4B76B000 00008DC0 PersonalityModuleBrandingDxe.efi 480BA000 0006D300 DellHiiBrowser.efi 4B701000 00069540 DellHiiFormsProcessor2Dxe.efi 4804A000 0006F320 SystemBoardDxe.efi 48041000 00008340 PlatformType.efi 4803A000 00006900 WatchdogTimer.efi 48028000 00011620 DriverHealthManagerDxe.efi 48019000 0000E2E0 TlsAuthConfigDxe.efi 48004000 000146A0 PciPlatform.efi 47DDE000 00021240 DisplayEngine.efi 47DC3000 0001A3E0 PlatDriOverrideDxe.efi 47DBA000 00008F80 DellHiiDefaults.efi 47DAC000 0000D6E0 DellHiiConfigDxe.efi 47D62000 000496A0 DellHiiFormDisplay.efi 47D4F000 00012120 DpiPlatformIdDxe.efi 47D32000 0001CAA0 PciHostBridge.efi 5DD87000 00011000 CpuCsrAccess.efi 47D2A000 000070A0 DellChipsetInfoDxe.efi 47D1F000 0000A580 DpiClocksDxe.efi 47D0D000 00011AE0 DpiSataDxe.efi 47CFB000 00011F40 DpiUsbDxe.efi 47CEC000 0000ECC0 DpiNdcDxe.efi 47CD3000 00018AC0 DellNvramDefaultAttributeDxe.efi 47CCB000 000075C0 DellPlatformAttributeDefaultsDxe.efi 5DD78000 0000F000 PchSpiRuntime.efi 5DD6A000 0000E000 PchResetRuntime.efi 47CC3000 00007A20 SmmAccess.efi 47CBC000 00006340 DellClocksDxe.efi 47CB3000 000082C0 DellPreviousBootErrorDxe.efi 47CA0000 00012A40 SbServiceProtocolDxe.efi 47C99000 000068C0 DellPlatformNvramDefaultsTableDxe.efi 5DD59000 00011000 FwBlockService.efi 5DD45000 00014000 PiSmmIpl.efi 47C90000 00008140 DellSpiFvbServicesDxe.efi 47C89000 000066E0 DpiTpmDxe.efi 5DD35000 00010000 VariableSmmRuntimeDxe.efi 47C7F000 000098E0 DellNvramAttributeSmmDxe.efi 47C76000 000086A0 DellSpiAccessDxe.efi 47C60000 000155C0 PlatformCpuPolicy.efi 47C52000 0000DAA0 LpcPlatform.efi 47944000 000BB940 SecurityStubDxe.efi 47C4A000 000070A0 LegacyRegion2.efi 47C16000 00033E40 PchInitDxe.efi 5DD21000 00014000 PlatformReset.efi 47C04000 00011200 AcpiVTD.efi 47931000 00012040 HeciInitDxe.efi 47913000 0001D340 DellFirmwareManagementDxe.efi 478FE000 00014720 DellPciDevice2Dxe.efi 478F5000 00008040 DellManufacturingModeDxe.efi 478E9000 0000B7E0 DellAciMDxe.efi 478E0000 00008520 DellIpmi.efi 478CC000 00013880 DellMainMenuSetupDxe.efi 478A0000 0002BE40 DellBootManagerDxe.efi 47884000 0001B400 DellHiiInternalSetup.efi 47879000 0000AF60 DellHiiBrowserData.efi 4785C000 0001CFE0 DellSmaJobProcess.efi 4784E000 0000DEE0 TlvInventoryDxe.efi 47846000 00007160 DellLegacyTableMemoryDxe.efi 4783E000 00007E20 DellPasswordInterfaceDxe.efi 47810000 0002D5E0 DellPostErrorMessageDxe.efi 475E7000 00016000 DellBiosInterfaceDxe.efi 47804000 0000BF00 HiiServiceTag.efi 475DD000 00009D00 DellIpmiResetToDefaultsDxe.efi 475C6000 00016900 DellFileExplorerDxe.efi 475AD000 00018D40 DellPmDataAcquisitionDxe.efi 4758B000 00021500 DellSmbiosTableBuildDxe.efi 47580000 0000A9A0 DellRci2BuildTable.efi 47579000 000065C0 DellWmiAcpiBuffer.efi 47570000 00008420 DellBiosDataWipe.efi 474DA000 00095380 DellCpldFirmwareManagementDxe.efi 474D0000 00009140 DellBootToDiagsDxe.efi 474BD000 00012300 DellEmbSata.efi 4742D000 0008F120 DellIdracSecureBootMgmt.efi 47423000 00009DC0 DellSetBayAndSlotDxe.efi 5DD10000 00011000 DellNvdimmDxe.efi 47411000 000118E0 DellAepSecurityDxe.efi 473FF000 00011CA0 DellRaidFmpDxe.efi 473F6000 00008CC0 DellCapsuleUpdateDxe.efi 473ED000 000084A0 DellHiiWorkloadProfile.efi 473DF000 0000D6E0 DellChipsetMemInfoDxe.efi 473D8000 00006E20 DpiIdrac.efi 473D1000 000064C0 DpiBdsDxe.efi 473B5000 0001BD60 DellEarlyVideoSplashDxe.efi 4737A000 0003A580 CpuMpDxe.efi 47373000 00006860 DellMemoryInterfaceMemoryDxe.efi 4736B000 00007EE0 DellIdracInitialize.efi 4735B000 0000F720 DellDlst2.efi 4734F000 0000B820 DellNdcDxe.efi 47348000 000069A0 DellSecurityLayerDxe.efi 4732F000 00018A40 DellIdrac.efi 47313000 0001B500 DellLcProvisioning.efi 4730B000 00007580 DellSmmCommBufferTableBuildDxe.efi 47301000 00009420 DellRipsDxe.efi 472F2000 0000E2E0 DellIdracSma.efi 472E1000 00010300 DpiCpldDxe.efi 5DCFF000 00011000 PcRtc.efi 5DCF3000 0000C000 MonotonicCounterRuntimeDxe.efi 5DCD9000 0001A000 CapsuleRuntimeDxe.efi 472D8000 00008CA0 EsrtDxe.efi 472C9000 0000E480 SoftSkuStatus.efi 472B9000 0000FA40 IsPlatformSupportWhea.efi 472A3000 00015B60 MePolicyInitDxe.efi 47279000 0002A000 IioInit.efi 47251000 00027DA0 DellDeviceLocation.efi 4723E000 00012B40 DellCpuInterfaceDxe.efi 47236000 00008000 DellPmQueryDxe.efi 47225000 00010640 DellPmDataStorageDxe.efi 5DCCC000 0000D000 CallingInterface.efi 471F2000 000329A0 DellTpm.efi 471D9000 00018C00 DellProcessCapsuleDxe.efi 4711E000 000BA620 HiiSecureBootCustom.efi 47117000 00006BC0 UFSDxe.efi 470FE000 00018FC0 DellNvdimmChipsetDxe.efi 470F1000 0000C1C0 LegacyBiosPlatform.efi 470E7000 00009680 WheaSupport.efi 47003000 000E4000 CrystalRidge.efi 46FE6000 0001CFC0 SpsDxe.efi 46FDC000 00009A80 DellTurboEnablePerCore.efi 46FD1000 0000A6C0 PerformanceControlDxe.efi 46FAC000 00024800 LegacyBiosDxe.efi 46F9E000 0000DF40 DellPciElogDxe.efi 46F96000 00007AA0 DpiCpuVrDxe.efi 46F86000 0000F040 PpmInitialize.efi 46F7F000 00006CC0 LegacyInterruptHookDxe.efi 46F77000 00007BE0 LegacyBiosReverseThunk.efi 46E93000 000E4000 JedecNvDimm.efi 46E88000 0000AA00 DellUsbCsmSwitch.efi 46E77000 00010620 DellTxt.efi 46E41000 00035780 AcpiPlatform.efi 46DA1000 0009F9A0 DellBdsDxe.efi 46D45000 0005B140 Unknown(stgsr) Intel(R) RSTe 5.5.5.1005 SATA Driver 46CE9000 0005B140 Unknown(stgsr) Intel(R) RSTe 5.5.5.1005 sSATA Driver 46CDC000 0000CAC0 S3NvramSave.efi 46CCB000 000100A0 ReserveMem.efi 46CBD000 0000D940 ArpDxe.efi 46CA7000 00015600 Dhcp4Dxe.efi 46C83000 00023BA0 Ip4Dxe.efi 46C6D000 000157E0 MnpDxe.efi 46C5A000 00012FC0 Mtftp4Dxe.efi 46C49000 00010C60 SnpDxe.efi 46C36000 00012FA0 Udp4Dxe.efi 46C1F000 00016180 Dhcp6Dxe.efi 46BF5000 000294C0 Ip6Dxe.efi 46BE0000 00014160 Mtftp6Dxe.efi 46BBC000 00023240 TcpDxe.efi 46BA8000 00013180 Udp6Dxe.efi 46B88000 0001FE80 UefiPxeBcDxe.efi 46A80000 001071E0 TlsDxe.efi 46A64000 0001B060 HttpDxe.efi 46A49000 0001A660 HttpBootDxe.efi 46A31000 00017960 DnsDxe.efi 46A22000 0000EC80 BiosVideoDxe.efi 46A18000 00009A20 ConPlatformDxe.efi 46A07000 00010F20 ConSplitterDxe.efi 469F5000 00011F60 TerminalDxe.efi 469E6000 0000EBE0 PartitionDxe.efi 469D8000 0000DF80 UdfDxe.efi 469CE000 00009560 VgaClassDxe.efi 469C3000 0000AF60 DiskIoDxe.efi 469AE000 000149A0 Fat.efi 469A3000 0000A7E0 SataController.efi 46993000 0000F380 AtaBusDxe.efi 4697C000 00016D40 AtaAtapiPassThruDxe.efi 4694F000 0002CEE0 PciBusDxe.efi 46947000 00007900 IsaAcpi.efi 4693B000 0000B240 IsaBusDxe.efi 46934000 00006F00 EnglishDxe.efi 46928000 0000B060 ScsiBus.efi 46913000 00014460 ScsiDisk.efi 46903000 0000F400 UsbKbDxe.efi 468F8000 0000AE00 UsbMouseDxe.efi 468DC000 0001BD80 XhciDxe.efi 468C9000 00012240 EhciDxe.efi 468BB000 0000DFE0 OhciDxe.efi 468A6000 00014140 UsbBusDxe.efi 46898000 0000DB80 UsbMassStorageDxe.efi 46889000 0000E140 DellCdDvdFileSystem.efi 46879000 0000F760 BootOptionCacheDxe.efi 4686D000 0000B180 DellUsbMouseAbsolutePointerDxe.efi 4685F000 0000DD20 IsaSerialDxe.efi 46827000 00038000 NvmExpressDxe.efi 467BA000 0006CB00 DellHiiTextDisplay.efi 46749000 000706E0 DellHiiGraphicDisplay.efi 46711000 00037DA0 RegularExpressionDxe.efi 46708000 000080A0 DellXMLDrv1.efi 466E6000 00021660 DellHiiConverter.efi 465DD000 001085E0 iDRAC_Hii.efi 465CF000 0000DF60 Unknown(gnjgs) G200eW Matrox Graphics UEFI Driver 464A1000 0012DBC0 RAIDCore.efi 46457000 00049800 b57undi.efi 4640D000 00049800 b57undi.efi 4637F000 0008D2C0 cxundi.efi 462F1000 0008D2C0 cxundi.efi 462DA000 00016400 Unknown(ngkkh) Avago EFI SAS Driver 45EFE000 003DB860 Unknown(ttwgg) Log Size: 41,794
Not sure where this comes from. But es2021 doesn't go past that.
Also, I have seen several errors logged with DIMM_A5 module:
Record: 136 Date/Time: 01/11/2020 01:01:50 Source: system Severity: Critical Description: Multi-bit memory errors detected on a memory device at location(s) DIMM_A5. Record: 106 Date/Time: 01/09/2020 23:21:01 Source: system Severity: Critical Description: Correctable memory error logging disabled for a memory device at location DIMM_A5. Record: 47 Date/Time: 01/08/2020 15:54:25 Source: system Severity: Critical Description: Multi-bit memory errors detected on a memory device at location(s) DIMM_A5.
I have tried es2024 and setting its IP manually and it seems that it indeed cannot reach the network. However, I do see the interfaces, but they don't seem to be having link
~ # ip address | grep state
~ #
~ # ip address
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eno3: <BROADCAST,MULTICAST> mtu 1500 qdisc mq qlen 1000
link/ether b0:26:28:f5:2c:04 brd ff:ff:ff:ff:ff:ff
3: eno1np0: <BROADCAST,MULTICAST> mtu 1500 qdisc mq qlen 1000
link/ether b0:26:28:f5:2c:06 brd ff:ff:ff:ff:ff:ff
inet 10.192.0.158/22 brd 10.192.3.255 scope global eno1np0
valid_lft forever preferred_lft forever
4: eno4: <BROADCAST,MULTICAST> mtu 1500 qdisc mq qlen 1000
link/ether b0:26:28:f5:2c:05 brd ff:ff:ff:ff:ff:ff
5: eno2np1: <BROADCAST,MULTICAST> mtu 1500 qdisc mq qlen 1000
link/ether b0:26:28:f5:2c:07 brd ff:ff:ff:ff:ff:ff
# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
~ # route add default gw 10.192.0.1
route: SIOCADDRT: Network is unreachable
~ # lspci | grep net
01:00.0 Ethernet controller: Broadcom Limited NetXtreme BCM5720 Gigabit Ethernet PCIe
01:00.1 Ethernet controller: Broadcom Limited NetXtreme BCM5720 Gigabit Ethernet PCIe
18:00.0 Ethernet controller: Broadcom Limited BCM57412 NetXtreme-E 10Gb RDMA Ethernet Controller (rev 01)
18:00.1 Ethernet controller: Broadcom Limited BCM57412 NetXtreme-E 10Gb RDMA Ethernet Controller (rev 01)@Papaul if you visually check the network card, do they have link? does the switch see them being up?
papaul@asw-a-codfw> show interfaces ge-6/0/12 descriptions Interface Admin Link Description ge-6/0/12 up up es2024
for es2021 i know already about it
Logical Vlan TAG MAC STP Logical Tagging
interface members limit state interface flags
ge-6/0/12.0 294912 untagged
private1-a-codfw 2017 294912 Forwarding untaggedI just saw this:
[35393.835622] tg3 0000:01:00.0 eno3: Link is up at 1000 Mbps, full duplex [35393.835634] tg3 0000:01:00.0 eno3: Flow control is off for TX and off for RX [35393.835636] tg3 0000:01:00.0 eno3: EEE is disabled [35393.835660] IPv6: ADDRCONF(NETDEV_CHANGE): eno3: link becomes ready [35397.323562] IPv6: ADDRCONF(NETDEV_UP): eno1np0: link is not ready
@Marostegui switch didn't learn any MAC address on that interface
papaul@asw-a-codfw> show ethernet-switching table interface ge-6/0/12 MAC database for interface ge-6/0/12 MAC database for interface ge-6/0/12.0
I have unloaded and then loaded again bnxt_en kernel module and I can see the main iface disappearing and then coming back. So I am not sure this is OS related, I believe there must be something weird with the switch/vlan or something?
[36855.676818] Broadcom NetXtreme-C/E driver bnxt_en v1.9.2 [36855.688230] bnxt_en 0000:18:00.0 eth0: Broadcom BCM57412 NetXtreme-E 10Gb Ethernet found at mem 9da10000, node addr b0:26:28:f5:2c:06 [36855.688241] bnxt_en 0000:18:00.0: 63.008 Gb/s available PCIe bandwidth (8 GT/s x8 link) [36855.688483] bnxt_en 0000:18:00.0 eno1np0: renamed from eth0 [36855.713617] bnxt_en 0000:18:00.1 eth0: Broadcom BCM57412 NetXtreme-E 10Gb Ethernet found at mem 9da00000, node addr b0:26:28:f5:2c:07 [36855.713627] bnxt_en 0000:18:00.1: 63.008 Gb/s available PCIe bandwidth (8 GT/s x8 link) [36855.715133] bnxt_en 0000:18:00.1 eno2np1: renamed from eth0
@Marostegui i double check the switch configuration for both es2020 and es2024 and the DNS files from https://gerrit.wikimedia.org/r/#/c/operations/dns/+/563323/ everything looks good. I did another test on es2021 when the server boots at the step of PXE boot the switch learned the MAC address
Vlan MAC MAC Age Logical
name address flags interface
private1-a-codfw b0:26:28:f5:28:02 D - ge-3/0/27.0
``
an just after the installation begins the MAC address goes away@Marostegui I will focus more on troubleshooting this on the NIC level on Monday. Since the 1GB and 10GB interfaces are on the same card, I will use the 10GB interface over the 1GB interface and see if if that helps. Have a great week end.
@Papaul we bought this hosts with 10G NICs just in case, but we can try with 1G at the moment (like the other existing hosts use). We are nowhere near NIC saturation on those hosts, so 1G is fine. Arzhel suggests that maybe there is an option on the BIOS to "disable" the 10G part of those NICs and leave them with just 1G enabled?
I have tried to look thru the BIOS to find a way to disable the 10G capability but I have found nothing.
On the installer shell I have tried removing the driver for the 10G card and leave only 1G which seems to work:
~ # modprobe -r bnxt_en
~ # ip address
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
3: eno3: <BROADCAST,MULTICAST> mtu 1500 qdisc mq qlen 1000
link/ether b0:26:28:f5:2c:04 brd ff:ff:ff:ff:ff:ff
5: eno4: <BROADCAST,MULTICAST> mtu 1500 qdisc mq qlen 1000
link/ether b0:26:28:f5:2c:05 brd ff:ff:ff:ff:ff:ffAfter that, I have given eno3 and IP via network/interfaces, however the installer keeps wanting to use eno1np0 which doesn't appear there anymore as it is unloaded after I removed the module, so the network keeps failing as eno1np0 doesn't exist.
@MoritzMuehlenhoff any idea where that is hardcoded?
For a test maybe disable Puppet on the install* servers and add
d-i netcfg/choose_interface select eno3
to the buster.cfg config file. If that fixes it, we can figure out a good way going forward.
Thanks @MoritzMuehlenhoff - I have merged https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/563975/ to make sure we install (or test) es2024 with stretch, as ideally we'll install stretch on those hosts.
Buster comes with 10.4 and we are not ready for that yet. There is no 10.1 for buster.
So I am going to modify stretch.cfg
The hack didn't work. It keeps choosing a different NIC than eno3, which unfortunately looks like the 10G one.
I tried a different approach which was uninstalling the kernel module BEFORE running the network check, and then I generated the interfaces file manually, so it would only pick eno3, but the installer now looks for eno1 which doesn't exist, as I uninstalled the kernel module bnxt_en.
~ # ip address
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
4: eno3: <BROADCAST,MULTICAST> mtu 1500 qdisc mq qlen 1000
link/ether b0:26:28:f5:2c:04 brd ff:ff:ff:ff:ff:ff
5: eno4: <BROADCAST,MULTICAST> mtu 1500 qdisc mq qlen 1000
link/ether b0:26:28:f5:2c:05 brd ff:ff:ff:ff:ff:ff~ # cat /etc/network/interfaces
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).
source /etc/network/interfaces.d/*
# The loopback network interface
auto lo
iface lo inet loopback
# The primary network interface
allow-hotplug eno1
iface eno1 inet static
address 10.192.0.158/22
gateway 10.192.0.1
# dns-* options are implemented by the resolvconf package, if installed
dns-nameservers 10.3.0.1
dns-search unassigned:(
Last test done with @ayounsi:
I have removed the drivers tg3 and bnxt_en from the OS:
/bin # ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
/bin # ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00However they still show as UP on the switch
Some more notes.
The only way to ping the gateway seems to be using the 1G ports, and making sure the 10G aren't there.
The 1G ifaces are controlled by tg3 module, so I have removed both, 1G and 10G drivers and then only loaded 1G one:
~ # modprobe -r bnxt_en ~ # modprobe -r tg3 ~ # modprobe tg3 ~ # ip addr add 10.192.0.158/22 dev eno3 ~ # ip link set eno3 up ~ # ping 10.192.0.1 PING 10.192.0.1 (10.192.0.1): 56 data bytes 64 bytes from 10.192.0.1: seq=0 ttl=64 time=1011.309 ms 64 bytes from 10.192.0.1: seq=1 ttl=64 time=11.212 ms 64 bytes from 10.192.0.1: seq=2 ttl=64 time=3.554 ms
The problem is that the installation keeps waiting for eno1 or eno2 which seem to be the 10G interfaces, and ignores the fact that eno3 is the one with link :(
I have been trying to rename the interfaces but I have not been able to do so, so far, as it is a Busybox shell.
I guess if we can force the installation to only consider eno3 it might work?
The 10G card is identical to what we have running fine on Stretch in e.g. ms-be2050 and I also validated there are no missing firmware packages etc, this seems entirely fine from the Linux end. Have we tried plugging the server into a different switch port or replaced the cable?
Thanks @MoritzMuehlenhoff for your tests.
So recap:
Ideas?
For what is worth, exactly the same behaviour is happening on es2020.
And what was done at T242481#5797643 for es2024 also works and the 1G NIC is able to ping its gateway.
No link on any of them:
~ # ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eno1: <BROADCAST,MULTICAST> mtu 1500 qdisc mq qlen 1000
link/ether b0:26:28:f5:28:04 brd ff:ff:ff:ff:ff:ff
3: eno2d1: <BROADCAST,MULTICAST> mtu 1500 qdisc mq qlen 1000
link/ether b0:26:28:f5:28:05 brd ff:ff:ff:ff:ff:ff
4: eno3: <BROADCAST,MULTICAST> mtu 1500 qdisc mq qlen 1000
link/ether b0:26:28:f5:28:02 brd ff:ff:ff:ff:ff:ff
5: eno4: <BROADCAST,MULTICAST> mtu 1500 qdisc mq qlen 1000
link/ether b0:26:28:f5:28:03 brd ff:ff:ff:ff:ff:ffForcing them to come up:
~ # ip link set eno1 up ~ # ip link set eno2d1 up ~ # ip link set eno3 up ~ # ip link set eno4 up
And they do come up:
~ # ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eno1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq qlen 1000
link/ether b0:26:28:f5:28:04 brd ff:ff:ff:ff:ff:ff
3: eno2d1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq qlen 1000
link/ether b0:26:28:f5:28:05 brd ff:ff:ff:ff:ff:ff
4: eno3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq qlen 1000
link/ether b0:26:28:f5:28:02 brd ff:ff:ff:ff:ff:ff
inet6 2620:0:860:101:b226:28ff:fef5:2802/64 scope global dynamic
valid_lft 2591981sec preferred_lft 604781sec
inet6 fe80::b226:28ff:fef5:2802/64 scope link
valid_lft forever preferred_lft forever
5: eno4: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq qlen 1000
link/ether b0:26:28:f5:28:03 brd ff:ff:ff:ff:ff:ffHowever, they yet don't get an IP from DHCP and installation fails and they seem down again:
~ # ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eno1: <BROADCAST,MULTICAST> mtu 1500 qdisc mq qlen 1000
link/ether b0:26:28:f5:28:04 brd ff:ff:ff:ff:ff:ff
3: eno2d1: <BROADCAST,MULTICAST> mtu 1500 qdisc mq qlen 1000
link/ether b0:26:28:f5:28:05 brd ff:ff:ff:ff:ff:ff
4: eno3: <BROADCAST,MULTICAST> mtu 1500 qdisc mq qlen 1000
link/ether b0:26:28:f5:28:02 brd ff:ff:ff:ff:ff:ff
5: eno4: <BROADCAST,MULTICAST> mtu 1500 qdisc mq qlen 1000
link/ether b0:26:28:f5:28:03 brd ff:ff:ff:ff:ff:ffI force them back up:
~ # ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eno1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq qlen 1000
link/ether b0:26:28:f5:28:04 brd ff:ff:ff:ff:ff:ff
3: eno2d1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq qlen 1000
link/ether b0:26:28:f5:28:05 brd ff:ff:ff:ff:ff:ff
4: eno3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq qlen 1000
link/ether b0:26:28:f5:28:02 brd ff:ff:ff:ff:ff:ff
inet6 2620:0:860:101:b226:28ff:fef5:2802/64 scope global dynamic
valid_lft 2591998sec preferred_lft 604798sec
inet6 fe80::b226:28ff:fef5:2802/64 scope link
valid_lft forever preferred_lft forever
5: eno4: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq qlen 1000
link/ether b0:26:28:f5:28:03 brd ff:ff:ff:ff:ff:ffI try to assign an IP to eno3 (1G NIC) and I can successfully ping its gw:
~ # ip addr add 10.192.0.157/22 dev eno3 ~ # ip link set eno3 up ~ # ping 10.192.0.1 PING 10.192.0.1 (10.192.0.1): 56 data bytes 64 bytes from 10.192.0.1: seq=0 ttl=64 time=0.896 ms 64 bytes from 10.192.0.1: seq=1 ttl=64 time=0.407 ms
I go back to the installation menu and if fails, as it tries to use eno1, which doesn't have an IP.
I give eno1 the same IP and it of course fails
2: eno1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq qlen 1000
link/ether b0:26:28:f5:28:04 brd ff:ff:ff:ff:ff:ff
inet 10.192.0.157/22 brd 10.192.3.255 scope global eno1
valid_lft forever preferred_lft forever
~ # route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 10.192.0.1 0.0.0.0 UG 0 0 0 eno1
10.192.0.0 0.0.0.0 255.255.252.0 U 0 0 0 eno1
~ # ping 10.192.0.1
PING 10.192.0.1 (10.192.0.1): 56 data bytes
^C
--- 10.192.0.1 ping statistics ---
6 packets transmitted, 0 packets received, 100% packet lossSo, I digged into this a little: Interface auto setup not working if one the devices has no link seems to be a known bug: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=700292
If we don't have a clean way to disable the 10G card until it's actually linked/used, we can try passing the MAC address of the 1G interface via BOOTIF=aa-bb-cc-dd-ee-ff in the pxelinux.cfg (https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=615600)
@Papaul can you double check (maybe even with the vendor) if there is a way to disable the 10G port for now?
BOOTIF doesn't seem to be making any difference, the host keeps failing.
While the option shows on the cmdline with the correct mac:
BOOT_IMAGE=debian-installer/amd64/linux initrd=debian-installer/amd64/initrd.gz vga=normal auto-install/enable=true preseed/url=http://apt.wikimedia.org/autoinstall/preseed.cfg DEBCONF_DEBUG=5 netcfg/choose_interface=auto netcfg/get_hostname=unassigned netcfg/get_domain=unassigned netcfg/dhcp_timeout=60 BOOTIF=b0-26-28-f5-28-02 --- console=ttyS1,115200n8
Which is eno3:
4: eno3: <BROADCAST,MULTICAST> mtu 1500 qdisc mq qlen 1000
link/ether b0:26:28:f5:28:02 brd ff:ff:ff:ff:ff:ff~ # ip addr add 10.192.0.157/22 dev eno3 ~ # ip link set eno3 up ~ # ping 10.192.0.1 PING 10.192.0.1 (10.192.0.1): 56 data bytes 64 bytes from 10.192.0.1: seq=0 ttl=64 time=1006.032 ms 64 bytes from 10.192.0.1: seq=1 ttl=64 time=5.930 ms 64 bytes from 10.192.0.1: seq=2 ttl=64 time=0.393 ms 64 bytes from 10.192.0.1: seq=3 ttl=64 time=0.428 ms
When I attempt to configure the network manually for the installing it keeps choosing eno1, which is the 10G without link.
I have found some bugs regarding BOOTIF, but doesn't seem to be affecting us, ie: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=826844
So adding the 01- did the trick and es2020 is installing:
append initrd=debian-installer/amd64/initrd.gz vga=normal auto-install/enable=true preseed/url=http://apt.wikimedia.org/autoinstall/preseed.cfg DEBCONF_DEBUG=5 netcfg/choose_interface=auto netcfg/get_hostname=unassigned netcfg/get_domain=unassigned netcfg/dhcp_timeout=60 BOOTIF=01-b0-26-28-f5-28-02 --- console=ttyS1,115200n8
This is a workaround, but we have to install 12 hosts in total (6 per DC), @Papaul @wiki_willy can we double check with Dell if (and how) the 10G can be disabled on BIOS or somewhere else?
Forgot to thank @MoritzMuehlenhoff for all the help and time with the troubleshooting
Indeed, 1G switch ports are RJ45, 10G are SFP. We could try to put an SFP-T on the server side. But I don't think it will work.
So I asked Dell if it was possible to replace the NIC card we have now with 2 separate NiC cards ( 1 x10Gb NiC and 1 x1GB NIC). He is checking with our account rep to see if it is possible and he will get back in touch with me.
Change 564724 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] install_server: Changing es2023 MAC to the 10G one
Change 564724 merged by Marostegui:
[operations/puppet@production] install_server: Changing es2023 MAC to the 10G one
pxelinux has a generic option to pass the MAC in place when receiving the boot image as BOOTIF: https://wiki.syslinux.org/wiki/index.php?title=SYSLINUX#SYSAPPEND_bitmask
Let's try a reimage os es2020 with
ipappend 2
added to /srv/tftpboot/stretch-installer/pxelinux.cfg/ttyS1-115200 (between the lines starting with "kernel" and "append initrd" to confirm.
Mentioned in SAL (#wikimedia-operations) [2020-01-14T16:26:17Z] <marostegui> Disable temporarily puppet on install1002 and install2002 - T242481
Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts:
es2020.codfw.wmnet
The log can be found in /var/log/wmf-auto-reimage/202001141627_marostegui_202850_es2020_codfw_wmnet.log.
Mentioned in SAL (#wikimedia-operations) [2020-01-14T16:41:24Z] <marostegui> Enable puppet back on install1002 and install2002 - T242481
Change 564729 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Pass down MAC address of to installing system via BOOTIF
This install error is caused by the fact that these servers have a dual port NIC with 1G and 10G interfaces, but only the 1G interface has a link currently (as we miss a 10G switch in that rack). This causes the error from this task, it's a long-standing bug in netcfg: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=700292
There's however a workaround: If the MAC address of the NIC from which d-i was PXE-booted is passed in the form BOOTIF=01-$MAC, then netcfg uses this MAC to determine the NIC. This was added in netcfg 1.63 (https://tracker.debian.org/news/504452/accepted-netcfg-163-source-i386/)
pxelinux has a generic option ("ipappend 2") to pass the MAC in place when receiving the boot image as BOOTIF down to d-i: https://wiki.syslinux.org/wiki/index.php?title=SYSLINUX#SYSAPPEND_bitmask
Valentín ran into a different issue last year which required BOOTIF and submitted a patch to d-i to prevent BOOTIF being passed to the installed system: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=921444
This change landed in Buster, but it's not available in Stretch.
It is however currently needed to install DB hosts which ATM need Stretch still. I did a codesearch for "BOOTIF" and there's a number of packages which parse/process BOOTIF. The presense of BOOTIF on the installed system is probably just cosmestic, but it's not easy to rule it out conclusively.
I see a number of options:
I'm inclined to go with 3.
I like option 3 indeed.
There is one more option which is a bit more painful, which is to live hack pxelinux and set the MAC address of the host we are going to install and do it manually. We have 6 hosts in eqiad and 12 in codfw.
Change 565237 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Install es2020 with stretch-installer-bootif TFTP environment
Change 565237 merged by Muehlenhoff:
[operations/puppet@production] Install es2020 with stretch-installer-bootif TFTP environment
Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts:
['es2020.codfw.wmnet']
The log can be found in /var/log/wmf-auto-reimage/202001161014_marostegui_138642.log.
Completed auto-reimage of hosts:
['es2020.codfw.wmnet']
Of which those FAILED:
['es2020.codfw.wmnet']
Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts:
['es2020.codfw.wmnet']
The log can be found in /var/log/wmf-auto-reimage/202001161221_marostegui_159400.log.
Completed auto-reimage of hosts:
['es2020.codfw.wmnet']
Of which those FAILED:
['es2020.codfw.wmnet']
Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts:
['es2020.codfw.wmnet']
The log can be found in /var/log/wmf-auto-reimage/202001161221_marostegui_159494.log.
Change 565265 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] install_server: Add stretch-installer-bootif to es20XX hosts
Change 565265 merged by Marostegui:
[operations/puppet@production] install_server: Add stretch-installer-bootif to es20XX hosts
So, this will be addressed in two parts:
Update from Dell after today's meeting.
They are looking for a way to see if it is possible and doable to rename the interfaces on the card so that the 1GB interface can show up as 1-1-1 and not 1-3-1.
They will get back in touch with me.
Last update from Dell below. Nothing we don't know about already.
I have not found the ability to disable a single port on the card. Only the whole card:
cid:image003.png@01D5CC83.D3EC79D0
There is no setting to disable single ports on the NDC (Network daughter card).
I will ask a few others to see if they have any additional ideas.
Thanks Papaul! I think we don't need to pursue the "let's disable the unused port" option further, the current solution within the debian-installer addresses this just fine (plus if we disable the 10G port, it'll cause further effort down the road to re-enable this once we have a 10G rack).
Change 564729 merged by Muehlenhoff:
[operations/puppet@production] Pass down MAC address of the installing system via BOOTIF on Buster
Change 573069 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] install_server: Pass bootif installer to new ES hosts
Change 573069 merged by Marostegui:
[operations/puppet@production] install_server: Pass bootif installer to new ES hosts