The link between cr2-magru and asw1-b3-magru failed today out of the blue.
Logs on the CR are full of these going back:
cmooney@cr2-magru> show log messages | match "et-0/0/1" Mar 4 07:30:01 cr2-magru fpc0 SMIC_PHY_DFE_TUNING: smic_phy_dfe_tuning_state: et-0/0/1 - DFE coarse/fine tuning completed (took 1996 ms); enabling DFE adaptive tuning. Mar 4 07:30:01 cr2-magru fpc0 MQSS(0): CMACPCS1: Cleared Ethernet MAC Local Fault Delta Event (et-0/0/1) Mar 4 07:30:01 cr2-magru fpc0 MQSS(0): CMACPCS1: Ethernet PCS Multilane Alignment Done Delta Event (et-0/0/1) Mar 4 07:30:01 cr2-magru fpc0 MQSS(0): CMACPCS1: Detected Ethernet MAC Local Fault Delta Event (et-0/0/1) Mar 4 07:30:01 cr2-magru fpc0 MQSS(0): CMACPCS1: Ethernet PCS Multilane Alignment Not Done Delta Event (et-0/0/1)
Light levels look reasonable-ish, though a little low on lane 2 on the CR:
cmooney@cr2-magru> show interfaces diagnostics optics et-0/0/1 | except "warn|alarm"
Mar 04 08:31:02
Physical interface: et-0/0/1
Module temperature : 33 degrees C / 91 degrees F
Module voltage : 3.2290 V
Lane 0
Laser bias current : 7.599 mA
Laser output power : 0.955 mW / -0.20 dBm
Laser receiver power : 0.205 mW / -6.88 dBm
Lane 1
Laser bias current : 7.199 mA
Laser output power : 0.815 mW / -0.89 dBm
Laser receiver power : 0.442 mW / -3.55 dBm
Lane 2
Laser bias current : 7.199 mA
Laser output power : 0.786 mW / -1.05 dBm
Laser receiver power : 0.070 mW / -11.52 dBm
Lane 3
Laser bias current : 7.199 mA
Laser output power : 0.910 mW / -0.41 dBm
Laser receiver power : 0.296 mW / -5.28 dBmThat level has been decresing steadily over the past while:
https://librenms.wikimedia.org/graphs/id=60583/type=sensor_dbm/from=1767256500/to=1772613300/
asw sides looks healthy:
cmooney@asw1-b3-magru> show interfaces diagnostics optics et-0/0/50 | except "warn|alarm"
Physical interface: et-0/0/50
Module temperature : 34 degrees C / 93 degrees F
Module voltage : 3.2560 V
Lane 0
Laser bias current : 7.199 mA
Laser output power : 0.785 mW / -1.05 dBm
Laser receiver power : 0.926 mW / -0.33 dBm
Lane 1
Laser bias current : 7.199 mA
Laser output power : 0.780 mW / -1.08 dBm
Laser receiver power : 0.871 mW / -0.60 dBm
Lane 2
Laser bias current : 7.199 mA
Laser output power : 0.806 mW / -0.94 dBm
Laser receiver power : 0.766 mW / -1.16 dBm
Lane 3
Laser bias current : 7.199 mA
Laser output power : 0.844 mW / -0.73 dBm
Laser receiver power : 0.868 mW / -0.61 dBmcr2 is reporting the fault is local:
cmooney@cr2-magru> show interfaces et-0/0/1
Mar 04 08:32:09
Physical interface: et-0/0/1, Enabled, Physical link is Down
Interface index: 151, SNMP ifIndex: 579
Description: Core: asw1-b3-magru:et-0/0/50 {#70130}
Link-level type: Ethernet, MTU: 9192, MRU: 9200, Speed: 100Gbps, BPDU Error: None, Loop Detect PDU Error: None, Ethernet-Switching Error: None, Loopback: Disabled,
Source filtering: Disabled, Flow control: Enabled
Pad to minimum frame size: Disabled
Device flags : Present Running Down
Interface Specific flags: Internal: 0x100200
Interface flags: Hardware-Down SNMP-Traps Internal: 0x4000
Link flags : None
CoS queues : 8 supported, 8 maximum usable queues
Schedulers : 0
Current address: b4:f9:5d:30:e1:43, Hardware address: b4:f9:5d:30:e1:43
Last flapped : 2026-03-04 08:26:23 UTC (00:05:46 ago)
Input rate : 0 bps (0 pps)
Output rate : 0 bps (0 pps)
Active alarms : LINK
Active defects : LINK, LOCAL-FAULTI think we probably want to try and replace the SFP here, probably starting with the CR2 side. Though a reseat and checking/cleaning the fibres might be another option (and the issue could well be the module in the asw either).