@ayounsi What day would work best for you to assist trouble shoot

Cleaned fiber replaced optic

Unfortunately the errors are still happening: https://librenms.wikimedia.org/graphs/to=1691387100/id=11592/type=port_errors/from=1690782300/

@ayounsi i will be in early tomorrow working on unboxing 15 pallets of servers that have arrived will you be available tomorrow to assist. I would like to try another optic

Yep, ping me.

We are monitoring this error it has been 12 days with no faults

Unfortunately the errors are back, even though not much it's still better to fix the issue.

Yeah I think we still need to look at this, further errors on the link today. Seems somewhat related to throughput, but we are miles away from capacity (peaks under 2Gb/sec).

I'd say worth trying an optic swap on one end, then the other to see if it fixes. May be cable but light levels look ok.

Icinga downtime and Alertmanager silence (ID=43516157-a1a8-45c7-82a5-d013fe5b4dda) set by cmooney@cumin1001 for 2:00:00 on 2 host(s) and their services with reason: replacing optics to troubleshoot errors on core switch link

lsw1-f1-eqiad.mgmt,ssw1-e1-eqiad.mgmt

Optic in ssw1-e1-eqiad et-0/0/8 was replaced, new one now working we should keep an eye and see if it fires again.

@cmooney i have not seen any new faults on this ticket. are you ok closing this ticket?

In T342502#9309447, @Jclark-ctr wrote:

@cmooney i have not seen any new faults on this ticket. are you ok closing this ticket?

Thanks yeah as Arzhel said the link looks clean now. We should mark the optic that was removed from ssw1-e1-eqiad as faulty. Maybe hold on to it for some emergency just in case as they're not cheap. Should we also speak to Rob about ordering another spare to keep our spare stocks up? Or what's the normal process there.

@ayounsi if you wouldn't mind messaging me a time that works best with you so we can fix this

Mentioned in SAL (#wikimedia-operations) [2023-12-08T14:44:48Z] <XioNoX> drain eqiad-codfw lumen transport for maintenance - T342502

cleaned both sides of cable and replaced optic if errors continue we will replace cable