Page MenuHomePhabricator

asw2-c-eqiad fpc3 Rear QSFP+ PIC Chan# 1 flapping
Closed, ResolvedPublic

Description

This syslog message has been flapping for quite a while:

Mar 11 20:10:23  asw2-c-eqiad fpc3 Rear QSFP+ PIC Chan# 1: Rx loss cleared
Mar 11 20:12:23  asw2-c-eqiad fpc3 Rear QSFP+ PIC Chan# 1: Rx loss set
Mar 11 20:15:23  asw2-c-eqiad fpc3 Rear QSFP+ PIC Chan# 1: Rx loss cleared
Mar 11 20:17:24  asw2-c-eqiad fpc3 Rear QSFP+ PIC Chan# 1: Rx loss set
Mar 11 20:18:24  asw2-c-eqiad fpc3 Rear QSFP+ PIC Chan# 1: Rx loss cleared
Mar 11 20:19:24  asw2-c-eqiad fpc3 Rear QSFP+ PIC Chan# 1: Rx loss set

Last time I contacted JTAC on a similar issue, they were not able to tell me which port it was.
FPC3 has 2 used QSFP+ ports (for VC links). We need to swap one of the cable, and if no success, the other one.

So first FPC3-1/0 - to FPC2, which is a 1M 40G DAC
Then FPC3-1/1 - to FPC5, which is a 5M 40G DAC

Due to them being VC link, we should first deactivate it, before swapping it, so please sync up with me first.

Thanks!

Event Timeline

ayounsi triaged this task as Normal priority.Mar 11 2019, 9:03 PM
ayounsi created this task.
Restricted Application added a project: Operations. · View Herald TranscriptMar 11 2019, 9:03 PM

Mentioned in SAL (#wikimedia-operations) [2019-04-01T17:55:39Z] <XioNoX> remove asw2-c-eqiad:et-3/1/2 from disabled interfaces - T218059

ayounsi closed this task as Resolved.Apr 1 2019, 6:15 PM

There was a VC link between FPC3 and FPC8 that was acting up/flapping long time ago.
As we already had quite too many VC links, I disabled as well but left it connected in case we needed to turn it on quicky.
Those logs were the fallouts of that, where a cable was plugged between the two members but not configured.

Chris unplugged that cable from both sides. Now to be monitored. If no more flaps we can remove it totally.

FYI, the QSFP links are made of 4*10G channels, so a log like Rear QSFP+ PIC Chan# 1: Rx loss cleared are incomplete as they says something is wrong with a channel on a port, but not which port...