Looking at the [[ https://logstash.wikimedia.org/goto/ef72247b935d67487002e11f7926682d | logs ]] some VC links have been flapping more or less regularly.
VC link errors statistics are not in LibreNMS, looking more closely at the VCs other interfaces (using `show virtual-chassis vc-port statistics extensive | match "FPC|Port|CRC alignment errors"`) show CRC errors:
asw2-a-eqiad:
fpc1:1/0 - CRC alignment errors: 31
**fpc1:1/1 - CRC alignment errors: 401 <---- matches interface flap logs** - increased since last counter clear (17)
fpc8:1/1 - CRC alignment errors: 374063 - increased since last counter clear (54730)
asw2-b-eqiad:
**fpc1:1/1 - CRC alignment errors: 55 <---- matches interface flap logs** - increased since last counter clear (2)
fpc3:1/1 - CRC alignment errors: 937
fpc6:1/0 - CRC alignment errors: 36
asw2-c-eqiad:
fpc3:1/0 - CRC alignment errors: 398877076 - increased since last counter clear (38926979)
**fpc5:1/3 - CRC alignment errors: 610 <---- matches interface flap logs** - increased since last counter clear (268)
fpc6:1/2 - CRC alignment errors: 3
fpc8:1/1 - CRC alignment errors: 998
asw2-d-eqiad:
fpc8:1/0 - CRC alignment errors: 71
As it's not possible to know if all those errors are still relevant, I cleared the counters and will check them after the break.
In the meantime we should replace the faulty optics that match interface flaps logs.
All of those are MMFs with [[ https://www.fs.com/products/36179.html | QSFP+-40G-SR4 ]] on each side.
The main offender is this one: https://netbox.wikimedia.org/dcim/cables/3391/ so we should start with it.
@Jclark-ctr do we have spares on site? When would be the earliest you could do it in January?