Page MenuHomePhabricator
Create Task
Maniphest T167261

Faulty link between cr2-codfw and cr1-eqdfw
Closed, ResolvedPublic
Actions

Description

The link between cr2-codfw and cr1-eqdfw is suffering some kind of packet loss.
Its OSPF metric has been bumped so traffic prefers the other link from codfw to eqdfw.

Both optics rx/tx power are within reasonable limits (even though cr1-eqdfw:xe-1/0/0 is slightly low). No interfaces errors.

@Papaul @faidon Does that circuit have an unique identifier? so we can ask CyrusOne to investigate it and run more extensive tests.

Event Timeline

ayounsi created this task.Jun 7 2017, 7:32 AM
Restricted Application added a project: SRE. · View Herald TranscriptJun 7 2017, 7:32 AM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Papaul added a comment.Jun 12 2017, 5:05 PM

port 5-6
reference on the patch panel, please see below

New Doc 2017-06-12_3.pdf349 KBDownload

faidon added a comment.Jun 12 2017, 5:07 PM

Found an old email from CyrusOne:

codfw-circuits.png (356×1 px, 50 KB)

ayounsi added a comment.Jun 12 2017, 6:31 PM

Ticket 830782 opened with CyrusOne

ayounsi added a comment.Jun 13 2017, 7:33 PM

Circuit identified and troubleshooting started by CyrusOne.

Stashbot subscribed.Jun 15 2017, 7:17 PM

Mentioned in SAL (#wikimedia-operations) [2017-06-15T19:17:24Z] <XioNoX> Re-enabled link between cr2-codfw and cr1-eqdfw - T167261

ayounsi closed this task as Resolved.Jun 15 2017, 8:05 PM

Circuit has been back to the same levels of traffic as before the issue for 1h, no more issues reported in Icinga or Smokeping.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits