Page MenuHomePhabricator

asw2-a-eqiad VC link down
Closed, ResolvedPublic

Description

The event that caused connectivity issue around 14:00UTC today was due to a faulty Virtual Chassis link between fpc1 and fpc3.

Instead of dying and re-converging cleanly, it flapped many time.

I'm not aware of any flapping protection for VC ports. Opened T201097 to have better monitoring.

We can also see that fpc1:1/0 shows as up, even though there are no neighbor.

ayounsi@asw2-a-eqiad> show virtual-chassis vc-port   
fpc1:
--------------------------------------------------------------------------
PIC / Port   Type      ID    Trunk  Status       Speed        Neighbor  ID  Interface
1/0         Configured         -1    Up           40000

fpc3:
--------------------------------------------------------------------------
PIC / Port   Type      ID    Trunk  Status       Speed        Neighbor  ID  Interface
1/1         Configured         -1    Down         40000

Please replace the link asap.

Event Timeline

ayounsi triaged this task as High priority.Aug 2 2018, 8:32 PM
ayounsi created this task.

If we need to swap this cable, we will need to order more 5M 40G QFSP+ cables. I only have 3M spares. I don't know if we want to use the Fiberstore brand....we had a few bad cables during the initial setup. This would be the 3rd one which seems like a lot to me.

We need to always have spares of the cables we use in production. Are spares being tracked somewhere?
As this one is urgent to replace and Fiberstore is quick, I'd say go with them as well.

RobH mentioned this in Unknown Object (Task).Aug 3 2018, 5:34 PM
Cmjohnson closed subtask Unknown Object (Task) as Resolved.Aug 8 2018, 5:46 PM

Received the new cables and swapped fpc1-fpc3