Page MenuHomePhabricator

FPC1 Failure on cr1-esams - take 2
Closed, ResolvedPublic

Description

That seems to be the exact same issue that we had almost 2 years ago, tracked in T351304: FPC1 Failure on cr1-esams

We did a full FPC replacement, so it's a bit worrying that it's happening again.

cr1-esams> show system alarms 
2 alarms currently active
Alarm time               Class  Description
2025-08-31 09:20:53 UTC  Major  FPC 1 Major Errors
2025-08-31 09:17:05 UTC  Minor  FPC 1 Minor Errors

Interfaces to the switches are down:

et-1/0/0        up    down Core: asw1-bw27-esams:et-0/0/48 {#30367}
et-1/0/1        up    down Core: asw1-by27-esams:et-0/0/48 {#30368}

And logs full of those:

Sep  1 08:00:06  re0.cr1-esams fpc1 SYSLOG: 33 messages lost, message queue overflowed.
Sep  1 08:00:07  re0.cr1-esams fpc1 cmic_phy_dfe_tuning_state (et-1/0/0): All channels are ready!
Sep  1 08:00:07  re0.cr1-esams fpc1 cmic_plat_dfe_coarse_tuning_stop: CMIC(1/0) stop DFE tuning for et-1/0/0
Sep  1 08:00:07  re0.cr1-esams fpc1 cmic_plat_dfe_coarse_tuning_start: CMIC(1/0) start DFE tuning for et-1/0/0
Sep  1 08:00:08  re0.cr1-esams fpc1 xfi2xaui_configure XFI2XAUI(0) enabling channel(s) for port 0
Sep  1 08:00:08  re0.cr1-esams fpc1 xfi2xaui_passthru_XLAUI: id=0 port=1 line/sys_loopback=0/0
Sep  1 08:00:08  re0.cr1-esams fpc1 xfi2xaui_passthru_anyChannel: id=0 pwr/egr/ingEn=0/0/1 portmask=0x1e0 (1e0/80e0)          wanMask=0x0 line/sys_loopback=0/0
Sep  1 08:00:08  re0.cr1-esams fpc1 xfi2xaui_picxo_is_not_lock: TIMEOUT id=0 picxo_ingMask=0x01e0                 picxo_egrMask=0x0000 dat=0x000001e0 zeroCnt=0

I'll open a JTAC case.

Related Objects

Event Timeline

ayounsi triaged this task as High priority.
Restricted Application added a subscriber: Aklapper. · View Herald Transcript

JTAC case 2025-0901-838306 opened.

Mentioned in SAL (#wikimedia-operations) [2025-09-01T19:23:44Z] <XioNoX> cr1-esams> request chassis fpc slot 1 offline - T403360

JTAC replied saying that a linecard reboot could fix the issue. I did it (see SAL above), so far still working fine, will check tomorrow morning.

ayounsi claimed this task.

Good news, it's still up.