Page MenuHomePhabricator

Power alarm flap on asw-d-codfw:et-7/0/52 channel 3
Closed, ResolvedPublic

Description

About 500 of those flaps per day:

fpc7 qsfp-7/0/52 Chan# 3: %-: Rx power low warning set
fpc7 qsfp-7/0/52 Chan# 3: %-: Rx power low warning cleared

We can see that Lane 3 is significantly different than the other 3.

asw-d-codfw> show interfaces diagnostics optics et-7/0/52    
[...]
Lane 0
  Laser receiver power                      :  0.643 mW / -1.92 dBm
Lane 1
  Laser receiver power                      :  0.630 mW / -2.00 dBm
Lane 2
  Laser receiver power                      :  0.565 mW / -2.48 dBm
Lane 3
  Laser receiver power                      :  0.116 mW / -9.35 dBm

That interface connects to cr2-codfw:et-0/2/1 and is the only member of ae2.
cr2-codfw doesn't have any free 40G ports so we can't add another member to ae2 the time to troubleshot it.

  1. Failover traffic to cr1-codfw by bumping its VRRP priority (or lowering cr2-codfw's)
  2. Verify no traffic goes over the faulty link
  3. One after the other and checking the power level at each time:
    1. Replace the optic on the asw side
    2. Replace the optic on the cr2 side
    3. Replace the fiber
  4. When the level is back to normal, set the VRRP priority to its normal value

@Papaul do you have spare QSFP optics and fiber?

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

Mentioned in SAL (#wikimedia-operations) [2017-08-30T16:13:24Z] <XioNoX> lowering vrrp priority for the cr2<->asw-d-codfw link - T174366

Mentioned in SAL (#wikimedia-operations) [2017-08-30T16:34:10Z] <XioNoX> start work on the cr2<->asw-d-codfw link - T174366

Mentioned in SAL (#wikimedia-operations) [2017-08-30T16:49:37Z] <XioNoX> rolling back vrrp priority change for the cr2<->asw-d-codfw link - T174366

ayounsi claimed this task.

Papaul replaced the optic on the switch side, levels back to normal:

> show interfaces diagnostics optics et-7/0/52 | match "Laser receiver power"    
    Laser receiver power                      :  0.575 mW / -2.40 dBm
    Laser receiver power                      :  0.603 mW / -2.19 dBm
    Laser receiver power                      :  0.640 mW / -1.94 dBm
    Laser receiver power                      :  0.607 mW / -2.17 dBm