Page MenuHomePhabricator
Create Task
Maniphest T148891

cp1052 ethernet link down 2016-10-22 14:11
Closed, ResolvedPublic
Actions

Description

This is a prod cache_text machine. I've depooled it explicitly from service for now.

14:11 < icinga-wm> PROBLEM - Host cp1052 is DOWN: PING CRITICAL - Packet loss = 100%

RAC console still works, can log in, host is still up. No ethernet link, with the following in dmesg:

[Sat Oct 22 14:11:16 2016] bnx2x 0000:01:00.0 eth0: NIC Link is Down

While I'm looking at dmesg, there's also notable temp threshold events in dmesg ~4 days ago, which reminds that we still have T125205 outstanding to monitor for thermal issues...

Related Objects

Event Timeline

BBlack created this task.Oct 22 2016, 2:33 PM
Restricted Application added a project: SRE. · View Herald TranscriptOct 22 2016, 2:33 PM
Restricted Application added subscribers: Southparkfan, Aklapper. · View Herald Transcript
BBlack added a comment.Oct 22 2016, 2:40 PM

Interface on asw-c-eqiad says down as well:

bblack@asw-c-eqiad> show interfaces xe-8/0/7 
Physical interface: xe-8/0/7, Enabled, Physical link is Down
  Interface index: 926, SNMP ifIndex: 795
  Description: cp1052
  Link-level type: Ethernet, MTU: 1514, Speed: 10Gbps, Duplex: Full-Duplex, BPDU Error: None, MAC-REWRITE Error: None, Loopback: Disabled,
  Source filtering: Disabled, Flow control: Enabled
  Device flags   : Present Running Down
  Interface flags: Hardware-Down SNMP-Traps Internal: 0x0
  Link flags     : None
  CoS queues     : 8 supported, 8 maximum usable queues
  Current address: 50:c5:8d:a8:2c:8a, Hardware address: 50:c5:8d:a8:2c:8a
  Last flapped   : 2016-10-22 14:10:39 UTC (00:27:55 ago)
  Input rate     : 0 bps (0 pps)
  Output rate    : 0 bps (0 pps)
  Active alarms  : LINK
  Active defects : LINK
  Interface transmit statistics: Disabled

  Logical interface xe-8/0/7.0 (Index 293) (SNMP ifIndex 0)
    Flags: Device-Down SNMP-Traps Encapsulation: ENET2
    Input packets : 10782202 
    Output packets: 95
    Protocol eth-switch
      Flags: None
BBlack added a comment.Oct 22 2016, 2:44 PM

Tried resetting the interface on the switch side with test interface xe-8/0/7 restart-auto-negotiation as well as commits of disable then re-enable of the interface, no change.

BBlack added a comment.Oct 22 2016, 2:52 PM

Tried re-setting from host-side software, and seems to have worked!

In response to ifconfig eth0 down, dmesg had new output:

[Sat Oct 22 14:46:17 2016] failed to kill vid 0081/0 for device eth0

Then in response to ifconfig eth0 up, dmesg gave:

[Sat Oct 22 14:46:36 2016] bnx2x 0000:01:00.0 eth0: using MSI-X  IRQs: sp 37  fp[0] 39 ... fp[15] 54
[Sat Oct 22 14:46:37 2016] bnx2x 0000:01:00.0 eth0: Warning: Unqualified SFP+ module detected, Port 0 from FINISAR CORP.    part number FTLX1471D3BCL   
[Sat Oct 22 14:46:37 2016] bnx2x 0000:01:00.0 eth0: NIC Link is Up, 10000 Mbps full duplex, Flow control: ON - receive & transmit
[Sat Oct 22 14:46:37 2016] 8021q: adding VLAN 0 to HW filter on device eth0

The ifconfig commands messed up the state of the network stuff on the host (e.g. killed def gw setting, etc), and the software might be in a messy state in general from being offline, so I've rebooted the host now.

Stashbot subscribed.Oct 22 2016, 2:52 PM

Mentioned in SAL (#wikimedia-operations) [2016-10-22T14:52:25Z] <bblack> rebooted cp1052 - T148891

Stashbot added a comment.Oct 22 2016, 3:02 PM

Mentioned in SAL (#wikimedia-operations) [2016-10-22T15:02:01Z] <bblack> repool cp1052 - T148891

BBlack closed this task as Resolved.Oct 22 2016, 3:05 PM
BBlack claimed this task.

Seems ok post-reboot, repooled.

BBlack reopened this task as Open.Oct 22 2016, 4:06 PM
BBlack added a subscriber: Joe.

It failed again:

15:32 < icinga-wm> PROBLEM - Host cp1052 is DOWN: PING CRITICAL - Packet loss = 100%

@Joe depooled again ~15:37. Leaving it depooled this time, since it's clearly not a one-off thing.

BBlack moved this task from Backlog to Varnish v4 on the Traffic board.Oct 22 2016, 5:12 PM
BBlack moved this task from Varnish v4 to General on the Traffic board.
• ema subscribed.Oct 23 2016, 1:38 PM
• ema mentioned this in T148976: Strongswan Icinga check: do not report issues about depooled hosts.Oct 24 2016, 2:59 PM
• ema added a comment.Feb 10 2017, 3:33 PM
[Fri Feb 10 15:01:57 2017] bnx2x 0000:01:00.0 eth0: Warning: Unqualified SFP+ module detected, Port 0 from FINISAR CORP.    part number FTLX1471D3BCL   
[Fri Feb 10 15:01:58 2017] bnx2x 0000:01:00.0 eth0: NIC Link is Up, 10000 Mbps full duplex, Flow control: ON - receive & transmit

I brought the system back up now (ifconfig eth0 down ; ifconfig eth0 up) and did a bit of puppet agent dance.

Let's keep an eye on the machine without repooling for now, as it's not exactly reliable.

• ema added a subscriber: • Cmjohnson.Feb 13 2017, 9:04 AM

That didn't last long:

[Sat Feb 11 03:37:44 2017] bnx2x 0000:01:00.0 eth0: NIC Link is Down

Also worth mentioning, /var/log/mcelog contains multiple instances of the following error:

STATUS 8c000043000800c0 MCGSTATUS 0
MCGCAP 1000c14 APICID 0 SOCKETID 0 
CPUID Vendor Intel Family 6 Model 45
Hardware event. This is not a software error.
MCE 0
CPU 0 BANK 8 
MISC 122100100010068c ADDR 160cbf6000 
TIME 1482798331 Tue Dec 27 00:25:31 2016
MCG status:
MCi status:
Corrected error
MCi_MISC register valid
MCi_ADDR register valid
MCA: MEMORY CONTROLLER MS_CHANNEL0_ERR
Transaction: Memory scrubbing error
MemCtrl: Corrected patrol scrub error

@Cmjohnson ideas on how to proceed? In its current state the system is just wasting power.

faidon reassigned this task from BBlack to • Cmjohnson.Feb 24 2017, 5:38 PM
• Cmjohnson added a subscriber: faidon.Feb 24 2017, 6:17 PM

@faidon, I will swap out the sfp+ ...that is the most typical culprit. Do we need to schedule downtime? or can I do anytime?

faidon added a comment.Feb 24 2017, 6:38 PM

I believe @ema has depooled it, so any time should be OK.

• ema added a comment.Feb 24 2017, 7:10 PM

@Cmjohnson yes the system is indeed depooled. Please go ahead whenever it is convenient for you. Thanks!

• Cmjohnson added a comment.Feb 27 2017, 3:32 AM

@ema the sfp has been replaced I see a link light now. LMK if that fixes the problem.

Stashbot added a comment.Feb 27 2017, 11:04 AM

Mentioned in SAL (#wikimedia-operations) [2017-02-27T11:04:45Z] <ema> rebooting cp1052 into kernel 4.4.2-3+wmf8 T148891

Stashbot added a comment.Feb 27 2017, 11:37 AM

Mentioned in SAL (#wikimedia-operations) [2017-02-27T11:37:54Z] <ema> cp1052 repooled T148891

• ema closed this task as Resolved.Feb 27 2017, 11:40 AM

@Cmjohnson looks good. Thanks!

• ema mentioned this in T160405: pybal stops logging.Mar 15 2017, 10:14 AM
• ema mentioned this in T162132: cp3003 network interface issues.Apr 4 2017, 11:50 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits