cr1-esams:fpc0 errors
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	ayounsi
	Sep 19 2023, 2:43 PM

Description

Logs are full of:

Sep 19 14:41:32  re0.cr1-esams fpc0 dispatch_event_handler(691): EA[0:0].disp[3] SECONDARY_TIMEOUT (PPE 5 Zone 19).
Sep 19 14:41:35  re0.cr1-esams fpc0 dispatch_event_handler(691): EA[0:0].disp[3] SECONDARY_TIMEOUT (PPE 5 Zone 19).
Sep 19 14:41:38  re0.cr1-esams fpc0 dispatch_event_handler(691): EA[0:0].disp[3] SECONDARY_TIMEOUT (PPE 5 Zone 19).

relevant doc: https://supportportal.juniper.net/s/article/Syslog-message-SECONDARY-TIMEOUT-or-PRIMARY-TIMEOUT?language=en_US

TLDR, FPC restart is the first step.

Details

Related Changes in Gerrit:

	Subject	Repo	Branch	Lines +/-
	Repool esams after maintenance	operations/dns	master	+0 -2
	Depool esams for cr1 maintenance	operations/dns	master	+2 -0

Customize query in gerrit

Event Timeline

ayounsi created this task.Sep 19 2023, 2:43 PM

ayounsi triaged this task as High priority.

Restricted Application added a project: Infrastructure-Foundations. · View Herald TranscriptSep 19 2023, 2:43 PM

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

RhinosF1 subscribed.Sep 19 2023, 3:12 PM

Maintenance_bot added a project: SRE.Sep 19 2023, 3:29 PM

ayounsi moved this task from Backlog to This quarter on the netops board.Sep 27 2023, 8:08 AM

Mentioned in SAL (#wikimedia-operations) [2023-11-15T13:55:56Z] <XioNoX> disable peering/transit on cr1-esams for linecard reboot - T346779

Mentioned in SAL (#wikimedia-operations) [2023-11-15T14:03:00Z] <XioNoX> reboot fpc0 on cr1-esams - T346779

Change 987782 had a related patch set uploaded (by Ayounsi; author: Ayounsi):

[operations/dns@master] Depool esams for cr1 maintenance

https://gerrit.wikimedia.org/r/987782

gerritbot added a project: Patch-For-Review.Jan 4 2024, 2:38 PM

cmooney subscribed.Jan 4 2024, 2:52 PM

Change 987782 merged by Ayounsi:

[operations/dns@master] Depool esams for cr1 maintenance

https://gerrit.wikimedia.org/r/987782

Mentioned in SAL (#wikimedia-operations) [2024-01-04T15:01:28Z] <XioNoX> depool esams for router work - T346779

Change 987732 had a related patch set uploaded (by Ayounsi; author: Ayounsi):

[operations/dns@master] Repool esams after maintenance

https://gerrit.wikimedia.org/r/987732

Mentioned in SAL (#wikimedia-operations) [2024-01-04T15:16:09Z] <XioNoX> drain esams-eqiad transport - T346779

Mentioned in SAL (#wikimedia-operations) [2024-01-04T15:26:55Z] <XioNoX> disable peering/transit on cr1-esams for linecard reboot - T346779

Mentioned in SAL (#wikimedia-operations) [2024-01-04T15:30:22Z] <XioNoX> reboot fpc0 on cr1-esams - T346779

Mentioned in SAL (#wikimedia-operations) [2024-01-04T15:37:10Z] <XioNoX> re-enable peering/transit on cr1-esams - T346779

Mentioned in SAL (#wikimedia-operations) [2024-01-04T15:38:57Z] <XioNoX> undrain esams-eqiad transport - T346779

Change 987732 merged by Ayounsi:

[operations/dns@master] Repool esams after maintenance

https://gerrit.wikimedia.org/r/987732

Mentioned in SAL (#wikimedia-operations) [2024-01-04T15:47:58Z] <XioNoX> repool esams - T346779

Error logs stopped showing up after the linecard reboot. Monitoring it for a bit before closing the task.

Maintenance_bot removed a project: Patch-For-Review.Jan 4 2024, 4:30 PM

All good.

cr1-esams:fpc0 errorsClosed, ResolvedPublicActions

Description

Details

Event Timeline

cr1-esams:fpc0 errors
Closed, ResolvedPublic
Actions