Page MenuHomePhabricator
Create Task
Maniphest T218059

asw2-c-eqiad fpc3 Rear QSFP+ PIC Chan# 1 flapping
Closed, ResolvedPublic
Actions

Description

This syslog message has been flapping for quite a while:

Mar 11 20:10:23  asw2-c-eqiad fpc3 Rear QSFP+ PIC Chan# 1: Rx loss cleared
Mar 11 20:12:23  asw2-c-eqiad fpc3 Rear QSFP+ PIC Chan# 1: Rx loss set
Mar 11 20:15:23  asw2-c-eqiad fpc3 Rear QSFP+ PIC Chan# 1: Rx loss cleared
Mar 11 20:17:24  asw2-c-eqiad fpc3 Rear QSFP+ PIC Chan# 1: Rx loss set
Mar 11 20:18:24  asw2-c-eqiad fpc3 Rear QSFP+ PIC Chan# 1: Rx loss cleared
Mar 11 20:19:24  asw2-c-eqiad fpc3 Rear QSFP+ PIC Chan# 1: Rx loss set

Last time I contacted JTAC on a similar issue, they were not able to tell me which port it was.
FPC3 has 2 used QSFP+ ports (for VC links). We need to swap one of the cable, and if no success, the other one.

So first FPC3-1/0 - to FPC2, which is a 1M 40G DAC
Then FPC3-1/1 - to FPC5, which is a 5M 40G DAC

Due to them being VC link, we should first deactivate it, before swapping it, so please sync up with me first.

Thanks!

Related Objects

Event Timeline

ayounsi triaged this task as Medium priority.Mar 11 2019, 9:03 PM
ayounsi created this task.
Restricted Application added a project: SRE. · View Herald TranscriptMar 11 2019, 9:03 PM
Stashbot added a comment.Apr 1 2019, 5:55 PM

Mentioned in SAL (#wikimedia-operations) [2019-04-01T17:55:39Z] <XioNoX> remove asw2-c-eqiad:et-3/1/2 from disabled interfaces - T218059

ayounsi closed this task as Resolved.Apr 1 2019, 6:15 PM

There was a VC link between FPC3 and FPC8 that was acting up/flapping long time ago.
As we already had quite too many VC links, I disabled as well but left it connected in case we needed to turn it on quicky.
Those logs were the fallouts of that, where a cable was plugged between the two members but not configured.

Chris unplugged that cable from both sides. Now to be monitored. If no more flaps we can remove it totally.

FYI, the QSFP links are made of 4*10G channels, so a log like Rear QSFP+ PIC Chan# 1: Rx loss cleared are incomplete as they says something is wrong with a channel on a port, but not which port...

ayounsi mentioned this in T252797: asw2-d1-eqiad:VCP failure.May 14 2020, 5:20 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits