Page MenuHomePhabricator

codfw-eqiad Zayo link is down (cr2-codfw:xe-5/0/1)
Closed, ResolvedPublic

Description

cr2-codfw:xe-5/0/1 is reported as physical link down since May 10 14:35:53Z. The respective eqiad interface, cr2-eqiad:xe-5/2/3, is marked as up. I raised this at 20:25Z with Zayo over the phone, their ticket ID is 1048927.

Event Timeline

faidon lowered the priority of this task from High to Medium.May 10 2016, 9:51 PM

Link up since 21:12:38Z. Waiting to hear from Zayo about the root cause and if it was on their side.

faidon changed the task status from Open to Stalled.May 12 2016, 11:51 PM

The issue recovered before they had a chance to investigate. I have been asked whether we are interested in an formal RFO (reason for outage), which is an investigation carried by a different team and takes around 10 days. I was told that this comes at no charge regardless of the outcome and hence have requested the RFO. I will update the task once I received that on our mailbox.

And here's the formal RFO:

This was down again for 48 hours with the same symptoms. I raised it again with Zayo, which got assigned the case TTN-0001073020. They dispatched a tech at both 2323 Bryan and 1649 W Frankford and ended up fixing the issue. The resolving comment was "Zayo found a faulty fiber jumper in our Carrollton node, jumper has been replaced and all service has restored". I'm not entirely convinced and I will monitor the situation for a reoccurence.