Page MenuHomePhabricator

asw2-a-eqiad <-> cr2-eqiad fiber issue
Closed, ResolvedPublic

Description

asw2-a-eqiad complains:

Oct  1 15:34:54  asw2-a-eqiad fpc7 sfp-7/0/46 link 46 SFP receive power low  warning cleared
Oct  1 15:36:22  asw2-a-eqiad fpc7 sfp-7/0/46 link 46 SFP receive power low  warning set
Oct  1 15:38:56  asw2-a-eqiad fpc7 sfp-7/0/46 link 46 SFP receive power low  warning cleared
Oct  2 06:43:31  asw2-a-eqiad fpc7 sfp-7/0/46 link 46 SFP receive power low  warning set
Oct  2 06:44:47  asw2-a-eqiad fpc7 sfp-7/0/46 link 46 SFP receive power low  warning cleared
Oct  2 06:48:35  asw2-a-eqiad fpc7 sfp-7/0/46 link 46 SFP receive power low  warning set
Oct  2 06:49:51  asw2-a-eqiad fpc7 sfp-7/0/46 link 46 SFP receive power low  warning cleared
Oct  2 08:04:25  asw2-a-eqiad fpc7 sfp-7/0/46 link 46 SFP receive power low  warning set

With the interface being:

faidon@asw2-a-eqiad> show interfaces descriptions xe-7/0/46          
Interface       Admin Link Description
xe-7/0/46       up    up   Core: cr2-eqiad:xe-4/0/0 {#3455} [10Gbps DF]
``

Diagnostics:

faidon@asw2-a-eqiad> show interfaces diagnostics optics xe-7/0/46 | match "rx|receive" 
    Receiver signal average optical power     :  0.0361 mW / -14.42 dBm
    Laser rx power high alarm                 :  Off
    Laser rx power low alarm                  :  Off
    Laser rx power high warning               :  Off
    Laser rx power low warning                :  On
    Laser rx power high alarm threshold       :  2.2387 mW / 3.50 dBm
    Laser rx power low alarm threshold        :  0.0229 mW / -16.40 dBm
    Laser rx power high warning threshold     :  1.7783 mW / 2.50 dBm
    Laser rx power low warning threshold      :  0.0363 mW / -14.40 dBm

So this probably needs a fiber or optic replacement. @ayounsi, I think I didn't see any alerts for this, so this sounds like an opportunity for adding more monitoring :)

Event Timeline

faidon triaged this task as High priority.Wed, Oct 2, 8:29 AM
faidon created this task.
Restricted Application added a project: Operations. · View Herald TranscriptWed, Oct 2, 8:29 AM
elukey added a subscriber: elukey.Wed, Oct 2, 8:32 AM
CDanis added a subscriber: CDanis.Wed, Oct 2, 2:20 PM
ayounsi assigned this task to Cmjohnson.Wed, Oct 2, 3:38 PM
ayounsi added a subscriber: Cmjohnson.

Related to T203719.

@Cmjohnson same as when there are interfaces errors, but here monitor for new:
sfp-7/0/46 link 46 SFP receive power low warning set in show log messages once replaced.

Mentioned in SAL (#wikimedia-operations) [2019-10-02T23:38:21Z] <XioNoX> disable cr2-eqiad:xe-4/0/0 - T234416

Mentioned in SAL (#wikimedia-operations) [2019-10-02T23:42:00Z] <XioNoX> enable cr2-eqiad:xe-4/0/0 - T234416

ayounsi closed this task as Resolved.Wed, Oct 2, 11:43 PM

Better!

ayounsi@asw2-a-eqiad> show interfaces diagnostics optics xe-7/0/46 | match "rx|receive" 
    Receiver signal average optical power     :  0.0741 mW / -11.30 dBm
    Laser rx power high alarm                 :  Off
    Laser rx power low alarm                  :  Off
    Laser rx power high warning               :  Off
    Laser rx power low warning                :  Off
    Laser rx power high alarm threshold       :  1.0000 mW / 0.00 dBm
    Laser rx power low alarm threshold        :  0.0100 mW / -20.00 dBm
    Laser rx power high warning threshold     :  0.7943 mW / -1.00 dBm
    Laser rx power low warning threshold      :  0.0126 mW / -19.00 dBm

I swapped both optics