Page MenuHomePhabricator
Create Task
Maniphest T267672

Interface errors on cr1-eqiad:xe-3/2/1
Closed, ResolvedPublic
Actions

Description

Now that T263019 is done, traffic is flowing on that interface, and revealing CRC errors:
https://librenms.wikimedia.org/graphs/id=11615/type=port_errors/

Faulty cable or optic. Feel free to replace without draining traffic away. Ping me if any issue.

Thanks

Event Timeline

ayounsi triaged this task as High priority.Nov 10 2020, 6:46 PM
ayounsi created this task.
Restricted Application added a project: SRE. · View Herald TranscriptNov 10 2020, 6:46 PM
ayounsi mentioned this in Unknown Object (Task).Nov 10 2020, 6:52 PM
ayounsi closed this task as Resolved.Nov 16 2020, 5:09 PM
ayounsi claimed this task.

This is not showing errors anymore.

ayounsi reopened this task as Open.Nov 18 2020, 8:29 AM

They're back...

Cmjohnson moved this task from Backlog to Hardware Failure / Troubleshoot on the ops-eqiad board.Nov 19 2020, 6:05 PM
ayounsi reassigned this task from ayounsi to Cmjohnson.Nov 20 2020, 9:54 AM

Didn't see it was assigned to me, oops.

Stashbot added a comment.Nov 24 2020, 4:01 PM

Mentioned in SAL (#wikimedia-operations) [2020-11-24T16:01:19Z] <cmjohnson1> replacing the sfp at cr1-eqiad xe-3/2/1 T267672

Cmjohnson added a comment.Nov 24 2020, 4:05 PM

I replaced the sfp+ at cr1-eqiad xe-3/2/1 cleared the interface statistics on that port. Let's leave this open a few days to see if anything changes.

Cmjohnson added a comment.Nov 25 2020, 8:19 PM

noticed this today

Statistics last cleared: 2020-11-24 16:04:52 UTC (1d 04:14 ago)

Input rate     : 1272 bps (1 pps)
Output rate    : 3536 bps (4 pps)
Input errors: 2, Output errors: 0
Active alarms  : None
Active defects : None
PCS statistics                      Seconds
  Bit errors                          8496
  Errored blocks                     29484
Cmjohnson added a comment.Dec 4 2020, 1:59 PM

@XioNoX it appears a new fiber will need to be run. I will get to this next week. Do we need to schedule any downtime? The total time offline is 1 minute (long enough for me to swap out fiber on both ends).

ayounsi added a comment.Dec 4 2020, 2:25 PM

No need to schedule anything. Thanks.

Cmjohnson added a comment.Dec 10 2020, 5:24 PM

replaced the cable, gave it the same cable number, removed the old fiber. Cleared the interface statistics on cr1.

Cmjohnson closed this task as Resolved.Dec 18 2020, 5:34 PM

No new errors since I replace the fiber. Resolving this

Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL