Page MenuHomePhabricator
Create Task
Maniphest T201095

asw2-a-eqiad VC link down
Closed, ResolvedPublic
Actions

Description

The event that caused connectivity issue around 14:00UTC today was due to a faulty Virtual Chassis link between fpc1 and fpc3.

Instead of dying and re-converging cleanly, it flapped many time.

I'm not aware of any flapping protection for VC ports. Opened T201097 to have better monitoring.

We can also see that fpc1:1/0 shows as up, even though there are no neighbor.

ayounsi@asw2-a-eqiad> show virtual-chassis vc-port   
fpc1:
--------------------------------------------------------------------------
PIC / Port   Type      ID    Trunk  Status       Speed        Neighbor  ID  Interface
1/0         Configured         -1    Up           40000

fpc3:
--------------------------------------------------------------------------
PIC / Port   Type      ID    Trunk  Status       Speed        Neighbor  ID  Interface
1/1         Configured         -1    Down         40000

Please replace the link asap.

Related Objects
Search...

StatusSubtypeAssignedTask
 Resolved• CmjohnsonT201095 asw2-a-eqiad VC link down
Unknown Object (Task)

Event Timeline

ayounsi created this task.Aug 2 2018, 8:32 PM
ayounsi triaged this task as High priority.
Restricted Application added a project: SRE. · View Herald TranscriptAug 2 2018, 8:32 PM
ayounsi mentioned this in T201097: Add virtual chassis port status alerting.Aug 2 2018, 8:37 PM
ayounsi updated the task description. (Show Details)
ayounsi added a project: netops.
faidon mentioned this in T201145: asw2-a-eqiad FPC5 gets disconnected every 10 minutes.Aug 3 2018, 9:04 AM
• Cmjohnson added a comment.Aug 3 2018, 2:38 PM

If we need to swap this cable, we will need to order more 5M 40G QFSP+ cables. I only have 3M spares. I don't know if we want to use the Fiberstore brand....we had a few bad cables during the initial setup. This would be the 3rd one which seems like a lot to me.

ayounsi added a comment.Aug 3 2018, 4:01 PM

We need to always have spares of the cables we use in production. Are spares being tracked somewhere?
As this one is urgent to replace and Fiberstore is quick, I'd say go with them as well.

ayounsi added a subscriber: RobH.Aug 3 2018, 4:02 PM
RobH mentioned this in Unknown Object (Task).Aug 3 2018, 5:34 PM
• Cmjohnson closed subtask Unknown Object (Task) as Resolved.Aug 8 2018, 5:46 PM
• Cmjohnson added a comment.Aug 8 2018, 6:08 PM

Received the new cables and swapped fpc1-fpc3

ayounsi closed this task as Resolved.Aug 13 2018, 8:28 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits