Page MenuHomePhabricator

cr3-knams:xe-0/1/3 down
Closed, ResolvedPublic

Description

Since the software upgrade xe-0/1/3 has been down.
It's one of the 3 bundled links between knams/esams so not critical but should be investigated quickly.

A bounce didn't help.

Light is good:

ayounsi@cr3-knams# run show interfaces diagnostics optics xe-0/1/3 
Physical interface: xe-0/1/3
    Laser output power                        :  1.4250 mW / 1.54 dBm
    Laser receiver power                      :  0.1672 mW / -7.77 dBm

Related Objects

StatusSubtypeAssignedTask
Resolvedayounsi

Event Timeline

ayounsi triaged this task as High priority.Feb 6 2020, 4:00 PM
ayounsi created this task.
Restricted Application added a subscriber: Aklapper. · View Herald Transcript

Other side doesn't receive the light though:

ayounsi@asw2-esams> show interfaces diagnostics optics xe-6/0/4   
Physical interface: xe-6/0/4
    Laser output power                        :  1.3540 mW / 1.32 dBm
    Laser receiver power                      :  0.0001 mW / -40.00 dBm
ayounsi added subscribers: RobH, mark, faidon.

As it matches the reboot of cr3-knams I'd say the optic on that side needs to be replaced. (maybe the power fluctuation damaged it?).

But as the optic looks fine on the CLI, maybe it's just the fiber that got un-seated (eg. because of the fan spinning at high speed during the reboot).

So we would need someone onsite either ways. @faidon or @mark to approve. Then @RobH for the remote hands part.

Please file a procurement task for Willy/Rob to execute on :)

ayounsi mentioned this in Unknown Object (Task).Feb 7 2020, 2:05 PM

Opened T244574.

RobH added a subtask: Unknown Object (Task).Feb 18 2020, 5:05 PM
RobH closed subtask Unknown Object (Task) as Resolved.Mar 30 2020, 10:57 PM

Mentioned in SAL (#wikimedia-operations) [2020-06-03T06:11:02Z] <XioNoX> cr3-esams> request vmhost reboot re1 (backup re) - T244497

Mentioned in SAL (#wikimedia-operations) [2020-06-03T06:18:24Z] <XioNoX> cr3-esams> request chassis routing-engine master switch - T244497

Mentioned in SAL (#wikimedia-operations) [2020-06-03T06:54:53Z] <XioNoX> failover vrrp to cr3-esams - T244497

Mentioned in SAL (#wikimedia-operations) [2020-06-03T06:56:20Z] <XioNoX> deactivate peering/transit BGP cr2-esams - T244497

The upgrade and reboot of cr3-knams didn't help.

Solved in T244574.