@BBlack @Dzahn I didn't get any direction with this but I assumed swapping the DAC is the appropriate attempted fix. Please let me know if this resolves the issue

Mentioned in SAL (#wikimedia-operations) [2018-09-05T18:30:52Z] <mutante> cp1080 - pooling again after T203194 appears fixed

@Cmjohnson Thank you. I don't really know if that was the cause but looking at monitoring now everything is green. So i repooled the server just now.

cp1085 hit this today, virtually identical in all respects with the sequence of events and kernel/log outputs, etc. That's 2/16 nodes, a little under 2 months apart. Not an epidemic, but maybe worth looking into. Most likely this is a kernel driver bug or firmware bug rather than bad hardware. A simple reboot brough cp1085 back into service afterwards.

Mentioned in SAL (#wikimedia-operations) [2018-11-06T23:55:49Z] <mutante> cp1084 - network went down, powercycled, probably T203194

@BBlack same symptoms on cp1084 just now...

bnxt_en 0000:3b:00.0 enp59s0f0: Error (timeout: 500) msg {0x51 0xe4f3}
bnxt_en 0000:3b:00.0 enp59s0f0: hwrm_ring_free cp failed. rc:-1

Change 482658 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] site: Keep lvs2010 as a spare system while T203194 is not solved

https://gerrit.wikimedia.org/r/482658

Change 482658 merged by Vgutierrez:
[operations/puppet@production] site: Keep lvs2010 as a spare system while T203194 is not solved

https://gerrit.wikimedia.org/r/482658

I suspect our bug is fixed by:

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=73f21c653f930f438d53eed29b5e4c65c8a0f906

which included in the official upstream Linux releases from 4.19-rc7 onwards. The bnxt_en driver has a lot of commit traffic though, so I'm not sure it's reasonable to backport this fix very far (it might inadvertently depend on many other commits). Maybe should look into whether other vendor kernels nearer to what we're running have backported it?

In T203194#4870205, @BBlack wrote:

I suspect our bug is fixed by:

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=73f21c653f930f438d53eed29b5e4c65c8a0f906

which included in the official upstream Linux releases from 4.19-rc7 onwards. The bnxt_en driver has a lot of commit traffic though, so I'm not sure it's reasonable to backport this fix very far (it might inadvertently depend on many other commits). Maybe should look into whether other vendor kernels nearer to what we're running have backported it?

Or we can ask the Broadcom maintainers whether they can provide a backport to 4.9.x, it's one of the officially support LTS kernel branches after all. And the Debian stretch kernel uses the bnxt_en driver without local patches.

Actually, it is already in the 4.9.y LTS/stable branch, here: https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?h=linux-4.9.y&id=b2be15bb02b961146177d49204de22df3dddd415

yeah, it's included as part of 4.9.134: https://cdn.kernel.org/pub/linux/kernel/v4.x/ChangeLog-4.9.134

Even better, then we can simply get the 4.9.144-1 kernel from stretch-proposed-updates and test whether that is the correct fix

Yeah. It's hard to "prove" whether we have this bug fixed other than running a supposed fix on the bnxt_en cp10 fleet for a while as a statistical test, but probably the sooner we start on that the better.

The 4.9.144-1 kernel is fully production-ready, the point releases for Debian are used to rebase the Stretch kernel to the latest set of 4.9.x bug fixes (although depending on the final date for Stretch 9.7 there might be one further update still).

Change 484181 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] aptrepo: add component/kernel-proposed-updates to stretch-wikimedia

https://gerrit.wikimedia.org/r/484181

Change 484181 merged by Vgutierrez:
[operations/puppet@production] aptrepo: add component/kernel-proposed-updates to stretch-wikimedia

https://gerrit.wikimedia.org/r/484181

Change 484199 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] cache: Add kernel-proposed-updates component for cp1075-99

https://gerrit.wikimedia.org/r/484199

Change 484199 merged by Vgutierrez:
[operations/puppet@production] cache: Add kernel-proposed-updates component for cp1075-99

https://gerrit.wikimedia.org/r/484199

Mentioned in SAL (#wikimedia-operations) [2019-01-14T15:02:18Z] <vgutierrez> upgrading kernel in cp1075 to 4.1.144-1 - T203194

Mentioned in SAL (#wikimedia-operations) [2019-01-14T15:33:48Z] <vgutierrez> rolling restart of cp1076-cp1090 to upgrade to kernel 4.9.144 - T203194

kernel upgraded successfully in cp1075-cp1090:

vgutierrez@cumin1001:~$ sudo cumin cp[1075-1090].eqiad.wmnet 'uname -v'
16 hosts will be targeted:
cp[1075-1090].eqiad.wmnet
Confirm to continue [y/n]? y
===== NODE GROUP =====
(16) cp[1075-1090].eqiad.wmnet
----- OUTPUT of 'uname -v' -----
#1 SMP Debian 4.9.144-1 (2018-12-30)

Mentioned in SAL (#wikimedia-operations) [2019-01-15T01:12:21Z] <mutante> cp1078 - bnxt_en - TX timeout detected - Host cp1078 is DOWN - powercycled via mgmt (T203194)

Thanks for handling cp1078 @Dzahn. It looks like 4.9.144 is also affected

so far we've seen crashes in the following servers (updated on 16/01/19):

• cp1078 (twice)
• cp1080
• cp1084
• cp1085
• cp1088

on the Dell community forum there is a post with several people reporting the same issue. The last one suggesting that Dell could be replacing hardware to fix the issue. Maybe it's time to open a support case with Dell?

The reports in that thread are for RHEL 7, which uses 3.10 as the base layer kernel (but with backports for all kinds of drivers, so it's hard to tell how that maps to out 4.9 kernel. One thing we could try is to test the 4.19.12-1~bpo9+1 kernel from stretch-backports. If it still fails in that version, we can easily report it to the upstream maintainers given that 4.19 is the latest LTS branch. Or we point Dell to the thread and ask them them swap the NICs to a known working 10G card.

Mentioned in SAL (#wikimedia-operations) [2019-01-16T07:36:13Z] <vgutierrez> powercycling cp1088 - T203194

cp1088 has been affected as well after the kernel upgrade

In T203194#4880054, @Vgutierrez wrote:

on the Dell community forum there is a post with several people reporting the same issue. The last one suggesting that Dell could be replacing hardware to fix the issue.

I think you're referring to the last comment there at present where a user reports:

Hello, In my case, I am having this issue in 8 servers ( same specs). We have a case opened with dell. They are proposing replace the mainboard. The card is broadcom.

I think this is probably just the standard-issue Dell answer for unknown hardware problems: replace the mainboard and see if it works. Given how widespread the problem is, this is unlikely to help unless they've identified that a specific batch of mainboards have a specific issue causing this.

Did we ever look at NIC firmware levels on these? (I don't remember)... it could be that it needs a combination of bugfixed firmware and the bugfixed Linux driver.

See also this email thread where Michael Chan (broadcom driver dev) asks for firmware level output, sees the same numbers we have on cp1088, and tells them to upgrade: https://www.spinics.net/lists/netdev/msg519478.html

Mentioned in SAL (#wikimedia-operations) [2019-01-16T16:45:35Z] <vgutierrez> upgrading NIC firmware on cp1075 - T203194

Mentioned in SAL (#wikimedia-operations) [2019-01-16T17:05:32Z] <vgutierrez> upgrading NIC firmware in cp1076 - T203194

Mentioned in SAL (#wikimedia-operations) [2019-01-16T17:22:17Z] <vgutierrez> rolling NIC firmware upgrade cp[1077-1080] - T203194

Mentioned in SAL (#wikimedia-operations) [2019-01-17T09:02:12Z] <vgutierrez> rolling NIC firmware upgrade cp[1081-1090] - T203194

firmware upgrade completed for all the affected systems.

Change 485044 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] pybal: check for discrepancies in the configured services

https://gerrit.wikimedia.org/r/485044

This is looking way better than the previous attempt. All the systems have been running 19 days without issues since the firmware upgrade:

vgutierrez@cumin1001:~$ sudo cumin cp[1075-1090].eqiad.wmnet "uptime |cut -d' ' -f 4-5"
16 hosts will be targeted:
cp[1075-1090].eqiad.wmnet
Confirm to continue [y/n]? y
===== NODE GROUP =====
(16) cp[1075-1090].eqiad.wmnet
----- OUTPUT of 'uptime |cut -d' ' -f 4-5' -----
19 days

Change 494671 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] Revert "cache: Add kernel-proposed-updates component for cp1075-99"

https://gerrit.wikimedia.org/r/494671

Change 494671 abandoned by Vgutierrez:
Revert "cache: Add kernel-proposed-updates component for cp1075-99"

Reason:
this needs an intermediate commit for proper cleaning

https://gerrit.wikimedia.org/r/494671

Change 494673 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] cache: get rid of the wikimedia-kernel-updates apt component (1/2)

https://gerrit.wikimedia.org/r/494673

Change 494674 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] cache: get rid of the wikimedia-kernel-updates apt component (2/2)

https://gerrit.wikimedia.org/r/494674

Change 494673 merged by Vgutierrez:
[operations/puppet@production] cache: get rid of the wikimedia-kernel-updates apt component (1/2)

https://gerrit.wikimedia.org/r/494673

Change 494674 merged by Vgutierrez:
[operations/puppet@production] cache: get rid of the wikimedia-kernel-updates apt component (2/2)

https://gerrit.wikimedia.org/r/494674

Change 494690 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] aptrepo: Get rid of the no longer needed component/kernel-proposed-updates

https://gerrit.wikimedia.org/r/494690

Change 494690 merged by Vgutierrez:
[operations/puppet@production] aptrepo: Get rid of the no longer needed component/kernel-proposed-updates

https://gerrit.wikimedia.org/r/494690

@Vgutierrez The firmware update on the NICs fixed this for good, right? Can we close this task?

Mentioned in SAL (#wikimedia-operations) [2020-03-02T07:22:57Z] <vgutierrez> upgrading NICs FW on lvs2008 - T196560 T203194