Page MenuHomePhabricator
Create Task
Maniphest T211715

Interface errors on cr1-codfw:xe-5/3/1
Closed, ResolvedPublic
Actions

Description

See https://librenms.wikimedia.org/graphs/id=8288/type=port_errors/

Optic most likely needs to be replaced, please sync up with me so I can drain the traffic beforehand.

Related Objects

Event Timeline

ayounsi created this task.Dec 11 2018, 7:30 PM
ayounsi triaged this task as High priority.
Restricted Application added a project: SRE. · View Herald TranscriptDec 11 2018, 7:30 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
ayounsi mentioned this in T211728: Outbound BGP graceful shutdown.Dec 11 2018, 9:49 PM
Stashbot subscribed.Dec 12 2018, 7:08 PM

Mentioned in SAL (#wikimedia-operations) [2018-12-12T19:08:54Z] <XioNoX> disable BGP to telia on cr1-codfw - T211715

Stashbot added a comment.Dec 12 2018, 7:16 PM

Mentioned in SAL (#wikimedia-operations) [2018-12-12T19:16:44Z] <XioNoX> re-enable BGP to telia on cr1-codfw - T211715

Papaul closed this task as Resolved.Dec 13 2018, 3:20 PM

This is complete

ayounsi reopened this task as Open.Dec 13 2018, 3:57 PM
ayounsi claimed this task.
ayounsi added a subscriber: Papaul.

Still seeing errors. Next step is to either replace the patch cable or contact Telia.

ayounsi reassigned this task from ayounsi to Papaul.Dec 14 2018, 10:59 PM

Let's try to replace the patch cable. Please sync up with me so I can drain traffic first.

Stashbot added a comment.Dec 19 2018, 4:59 PM

Mentioned in SAL (#wikimedia-operations) [2018-12-19T16:59:29Z] <XioNoX> deactive BGP sessions to telia on cr1-codfw - T211715

Stashbot added a comment.Dec 19 2018, 7:28 PM

Mentioned in SAL (#wikimedia-operations) [2018-12-19T19:28:39Z] <XioNoX> reactive BGP sessions to telia on cr1-codfw - T211715

ayounsi closed this task as Resolved.Dec 19 2018, 10:55 PM
ayounsi claimed this task.

After many back and forth by email the issue has been fixed:

Dear customer,
After having our transmission vendor troubleshooting the DWDM section they identified the next problem:
“Issue identified on the link gain and was resolved.
Found all the SIMs L1 ports on CRL-CY nodes have RX error for some time.
Target Gain on the RCN-DB chassis 4 slot 9 has been changed from 11 to 13 and the issue seems to be resolved”
Could you please clear the counters and retest again for errors? We recently did the same on our side, so far the interface shows no errors.

ayounsi mentioned this in T222967: Interface errors on cr1-codfw: xe-5/3/1.May 10 2019, 4:49 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits