FPC1 Failure on cr1-esams
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	cmooney
	Nov 15 2023, 1:16 PM

Project Tags

Referenced Files

	F41507770: online.txt
	Nov 15 2023, 1:42 PM

Subscribers

Aklapper

ayounsi

cmooney

Description

FPC1 on cr1-esams has failed completely it seems:

cmooney@re0.cr1-esams> show system alarms  
2 alarms currently active
Alarm time               Class  Description
2023-11-10 06:41:34 UTC  Major  FPC 1 Major Errors
2023-11-10 06:41:22 UTC  Minor  FPC 1 Minor Errors

Log full of messages relating

Nov 15 13:00:01  re0.cr1-esams fpc1 xfi2xaui_picxo_is_not_lock: TIMEOUT id=0 picxo_ingMask=0x000f                 picxo_egrMask=0x0000 dat=0x0000000f zeroCnt=0
Nov 15 13:00:01  re0.cr1-esams fpc1 cmic_phy_dfe_tuning_state (et-1/0/0): All channels are ready!
Nov 15 13:00:01  re0.cr1-esams fpc1 cmic_plat_dfe_coarse_tuning_stop: CMIC(1/0) stop DFE tuning for et-1/0/0
Nov 15 13:00:01  re0.cr1-esams fpc1 cmic_plat_dfe_coarse_tuning_start: CMIC(1/0) start DFE tuning for et-1/0/0
Nov 15 13:00:02  re0.cr1-esams fpc1 xfi2xaui_configure XFI2XAUI(0) enabling channel(s) for port 0
Nov 15 13:00:02  re0.cr1-esams fpc1 xfi2xaui_passthru_XLAUI: id=0 port=1 line/sys_loopback=0/0
Nov 15 13:00:02  re0.cr1-esams fpc1 xfi2xaui_passthru_anyChannel: id=0 pwr/egr/ingEn=0/0/1 portmask=0x1e0 (1e0/80e0)          wanMask=0x0 line/sys_loopback=0/0
Nov 15 13:00:02  re0.cr1-esams fpc1 SYSLOG: 33 messages lost, message queue overflowed.
Nov 15 13:00:04  re0.cr1-esams fpc1 xfi2xaui_picxo_is_not_lock: TIMEOUT id=0 picxo_ingMask=0x000f                 picxo_egrMask=0x0000 dat=0x0000000f zeroCnt=0
Nov 15 13:00:04  re0.cr1-esams fpc1 cmic_phy_dfe_tuning_state (et-1/0/0): All channels are ready!
Nov 15 13:00:04  re0.cr1-esams fpc1 cmic_plat_dfe_coarse_tuning_stop: CMIC(1/0) stop DFE tuning for et-1/0/0
Nov 15 13:00:04  re0.cr1-esams fpc1 cmic_plat_dfe_coarse_tuning_start: CMIC(1/0) start DFE tuning for et-1/0/0
Nov 15 13:00:05  re0.cr1-esams fpc1 xfi2xaui_configure XFI2XAUI(0) enabling channel(s) for port 0
Nov 15 13:00:05  re0.cr1-esams fpc1 xfi2xaui_passthru_XLAUI: id=0 port=1 line/sys_loopback=0/0
Nov 15 13:00:05  re0.cr1-esams fpc1 xfi2xaui_passthru_anyChannel: id=0 pwr/egr/ingEn=0/0/1 portmask=0x1e0 (1e0/80e0)          wanMask=0x0 line/sys_loopback=0/0
Nov 15 13:00:05  re0.cr1-esams fpc1 SYSLOG: 33 messages lost, message queue overflowed.
Nov 15 13:00:07  re0.cr1-esams fpc1 cmic_phy_dfe_tuning_state: et-1/0/0 - DFE coarse/fine tuning completes; enabling DFE adaptive tuning.
Nov 15 13:00:07  re0.cr1-esams fpc1 cmic_phy_dfe_tuning_state (et-1/0/0): All channels are ready!

This means our links to the switches in esams are currently down:

cmooney@re0.cr1-esams> show interfaces descriptions | match "-1/" 
et-1/0/0        up    down Core: asw1-bw27-esams:et-0/0/48 {#30367}
et-1/0/1        up    down Core: asw1-by27-esams:et-0/0/48 {#30368}

First step is a reboot of the card and we take it from there.

Details

Related Changes in Gerrit:

	Subject	Author	Repo	Branch	Lines +/-
	De-pool esams due to edge router offline status	Cathal Mooney	operations/dns	master	+1 -0

Customize query in gerrit

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Resolved		ayounsi	T351304 FPC1 Failure on cr1-esams
					Unknown Object (Task)

Event Timeline

cmooney created this task.Nov 15 2023, 1:16 PM

cmooney triaged this task as High priority.

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 15 2023, 1:16 PM

Mentioned in SAL (#wikimedia-operations) [2023-11-15T13:17:06Z] <topranks> resetting FPC1 card in cr1-esams which has a major error and gone offline (T351304)

Reset completed, the card came back up briefly but quickly failed again

cmooney@re0.cr1-esams> show chassis fpc 1 detail 
Slot 1 information:
  State                               Online    
  Temperature                      38 degrees C / 100 degrees F
  Total CPU DRAM                 3584 MB
  Total RLDRAM                    259 MB
  Total DDR DRAM                 20352 MB
  FIPS Capable                        False 
  FIPS Mode                           False 
  Start time                          2023-11-15 13:19:12 UTC
  Uptime                              9 minutes, 47 seconds
  Max MPC base power consumption   376 Watts
  Max MIC0 power consumption        18 Watts
  Max MIC1 power consumption         0 Watts
  Max MPC total power consumption  394 Watts

cmooney@re0.cr1-esams> show system alarms 
2 alarms currently active
Alarm time               Class  Description
2023-11-15 13:23:00 UTC  Major  FPC 1 Major Errors
2023-11-15 13:22:51 UTC  Minor  FPC 1 Minor Errors

cmooney@re0.cr1-esams> show interfaces descriptions | match "et-1/" 
et-1/0/0        down  down Core: asw1-bw27-esams:et-0/0/48 {#30367}
et-1/0/1        down  down Core: asw1-by27-esams:et-0/0/48 {#30368}

Will need to raise a JTAC case.

Some logs following the issue of the "request chassis fpc online slot 1" command:

online.txt207 KBDownload

JTAC case 2023-1115-011066 opened.

ayounsi claimed this task.Nov 15 2023, 1:48 PM

Change 974539 had a related patch set uploaded (by Cathal Mooney; author: Cathal Mooney):

[operations/dns@master] De-pool esams due to edge router offline status

https://gerrit.wikimedia.org/r/974539

Change 974539 abandoned by Cathal Mooney:

[operations/dns@master] De-pool esams due to edge router offline status

Reason:

dup of other patch to do same thing