Page MenuHomePhabricator

Interface errors on cr2-eqiad:xe-4/0/0
Closed, ResolvedPublic

Description

See. https://librenms.wikimedia.org/device/device=2/tab=port/port=6828/

Input errors: Errors: 12, Drops: 0, Framing errors: 12

Please sync up with netops to replace its optics one one side and clear the counters clear interfaces statistics xe-4/0/0
Then monitor to verify if the issue is fixed.
If not, please repeat with the other side.

Event Timeline

ayounsi triaged this task as High priority.Sep 6 2018, 10:17 PM
ayounsi created this task.
Restricted Application added a project: Operations. · View Herald TranscriptSep 6 2018, 10:17 PM
Cmjohnson moved this task from Backlog to Up next on the ops-eqiad board.Sep 12 2018, 4:28 PM

Mentioned in SAL (#wikimedia-operations) [2018-09-13T17:25:02Z] <XioNoX> disable cr2:xe-4/0/0 (to asw-a) for optics replacement - T203719

Mentioned in SAL (#wikimedia-operations) [2018-09-13T17:29:19Z] <XioNoX> enable cr2:xe-4/0/0 (to asw-a) for optics replacement - T203719

Mentioned in SAL (#wikimedia-operations) [2018-09-13T17:31:37Z] <XioNoX> disable cr2:xe-4/0/0 (to asw-a) for optics replacement (round 2, 1st one didn't clear the errors, need to do the other side) - T203719

Mentioned in SAL (#wikimedia-operations) [2018-09-13T17:38:49Z] <XioNoX> enable cr2:xe-4/0/0 (to asw-a) - T203719

ayounsi closed this task as Resolved.Sep 13 2018, 5:44 PM

Lot better, thanks!

ayounsi reopened this task as Open.Oct 17 2018, 7:52 AM

Re-opening as we're seeing errors again (at a lower rate, but errors nonetheless)

https://librenms.wikimedia.org/graphs/to=1536910800/id=6828/type=port_errors/from=1536219600/ were the previous ones
https://librenms.wikimedia.org/graphs/to=1539762300/id=6828/type=port_errors/from=1539157500/ are the new less frequent errors

Need to wait longer between optics replacement as the errors are less frequent.

cr2-eqiad> show interfaces xe-4/0/0 extensive | match error 
 Input errors:
   Errors: 345, Drops: 0, Framing errors: 345
CRC/Align errors                       345

Mentioned in SAL (#wikimedia-operations) [2018-10-22T17:20:34Z] <XioNoX> disable cr2:xe-4/0/0 (to asw-a) for optics replacement - T203719

Mentioned in SAL (#wikimedia-operations) [2018-10-22T17:23:44Z] <XioNoX> enable cr2:xe-4/0/0 (to asw-a) for optics replacement - T203719

I replaced the optics on asw-a-eqiad:xe-2/1/2 {#3455} and cleaned the fiber.

ayounsi closed this task as Resolved.Oct 24 2018, 1:11 PM

Seems all solved.