Page MenuHomePhabricator
Create Task
Maniphest T244944

cr1-eqsin routing engine crashlooping after JunOS upgrade
Closed, ResolvedPublic
Actions

Description

cr1-eqsin was upgraded to 17.3R3-S6.3. After the upgrade, the routing engine never completes loading all routes over BGP, crashing partway through (after about 540k routes).

Due to lack of redundancy, eqsin is currently depooled, will stay that way until this is fixed.

JTAC case being opened.

Event Timeline

CDanis created this task.Feb 12 2020, 1:07 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 12 2020, 1:07 AM
ayounsi added a comment.Feb 12 2020, 1:08 AM

Opened JTAC Service Request ID: 2020-0211-0750

ayounsi added a comment.Feb 12 2020, 1:50 AM

Attached RSI and /var/log/ as well as replied to their initial questions.

Stashbot added a comment.Feb 12 2020, 12:53 PM

Mentioned in SAL (#wikimedia-operations) [2020-02-12T12:53:02Z] <XioNoX> cr1-eqsin RE failover - T244944

ayounsi added a comment.Feb 12 2020, 12:57 PM

This is a known bug (in JTAC recommended), and need to upgrade to the next S release (S7).

Stashbot added a comment.Feb 12 2020, 1:22 PM

Mentioned in SAL (#wikimedia-operations) [2020-02-12T13:22:33Z] <XioNoX> cr1-eqsin RE failover (final) - T244944

Stashbot added a comment.Feb 12 2020, 1:36 PM

Mentioned in SAL (#wikimedia-operations) [2020-02-12T13:36:32Z] <XioNoX> re-enable transit/peering on cr1-eqsin - T244944

ayounsi closed this task as Resolved.Feb 12 2020, 1:40 PM

Looks solved.

Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL