About 500 of those flaps per day:
fpc7 qsfp-7/0/52 Chan# 3: %-: Rx power low warning set
fpc7 qsfp-7/0/52 Chan# 3: %-: Rx power low warning cleared
We can see that Lane 3 is significantly different than the other 3.
asw-d-codfw> show interfaces diagnostics optics et-7/0/52 [...] Lane 0 Laser receiver power : 0.643 mW / -1.92 dBm Lane 1 Laser receiver power : 0.630 mW / -2.00 dBm Lane 2 Laser receiver power : 0.565 mW / -2.48 dBm Lane 3 Laser receiver power : 0.116 mW / -9.35 dBm
That interface connects to cr2-codfw:et-0/2/1 and is the only member of ae2.
cr2-codfw doesn't have any free 40G ports so we can't add another member to ae2 the time to troubleshot it.
- Failover traffic to cr1-codfw by bumping its VRRP priority (or lowering cr2-codfw's)
- Verify no traffic goes over the faulty link
- One after the other and checking the power level at each time:
- Replace the optic on the asw side
- Replace the optic on the cr2 side
- Replace the fiber
- When the level is back to normal, set the VRRP priority to its normal value
@Papaul do you have spare QSFP optics and fiber?