Page MenuHomePhabricator

cr2-esams rpd failure after enabling bgp 'graceful-shutdown' (June 2026)
Closed, ResolvedPublic

Description

As part of regular operations yesterday I was draining traffic off cr2-esams, so we could reset "pic 0" to reconfigure the speeds of the QSFP ports in the device after a change.

As part of the normal process we use for this I issued the below command to enable the BGP "graceful shutdown" feature, which lowers local-preference for all BGP learnt routes on the box, and adds the well-known "gshut" community to any it is announcing to signal to peers not to use it.

set protocols bgp graceful-shutdown sender

For whatever reason a minute or so later (did not appear to be right away) rpd (routing-protocol daemon, the service that controls routing protocols on JunOS) crashed:

Jun 15 15:00:10  cr2-esams kernel: pid 20071 (rpd), jid 0, uid 0: exited on signal 6 (core dumped)

Opening JTAC case with Juniper to see if they can explain this and if it is a known bug. Something similar may have occurred last week on cr2-drmrs on the same JunOS version, however there things recovered by themselves and quicker. But there were some unexplained ping losses to the device during that maintenance I didn't expect, and the same command had been issued.

Event Timeline

cmooney triaged this task as Low priority.

Juniper have come back to say this is known bug, somewhat expected I guess.

After decoding the coredump, it was confirmed that you are hitting an Internal PR behavior, where a Heap corruption causes the RPD crash when "graceful-shutdown sender" is enabled.

Fixed in 23.4R2-S8 which I guess we'll need to upgrade to with a whole lot of old school policy change stuff.

@Papaul FYI for any upgrades next week let's use 23.4R2-S8, older versions have this nasty bug.

Also recording JTAC case number is 2026-0616-761841

Probably a little premature to close. I've opened ticket 2026-0616-761841 to confirm exactly what JunOS versions are affected by this bug.

We'll likely need to upgrade any that are affected. I paused before resetting PIC 0 on cr3-eqsin to change a QSFP port speed as I wasn't sure if it was safe to use 'graceful-shutdown' or if we needed to use a more complicated method. If it turns out the latter is needed we may as well upgrade that router again when we drain it.

cmooney mentioned this in Unknown Object (Task).Jul 2 2026, 1:17 PM

FWIW I upgraded cr3-ulsfo as I had to drain it to reset the PIC and it was on known-bad release 23.4R2-S7.

Icinga downtime and Alertmanager silence (ID=bfbdc098-dd99-4c7c-917e-adc6a5d0016b) set by cmooney@cumin1003 for 2:00:00 on 5 host(s) and their services with reason: upgrade JunOS cr3-eqsin

cr1-codfw,cr[2-3]-eqsin,cr3-eqsin IPv6,cr3-eqsin.mgmt

Icinga downtime and Alertmanager silence (ID=18b1690e-ff0b-4bf2-96ef-198b1f9bd984) set by cmooney@cumin1003 for 2:00:00 on 3 host(s) and their services with reason: upgrade JunOS cr3-eqsin

lvs[5004-5006].eqsin.wmnet

Mentioned in SAL (#wikimedia-operations) [2026-07-07T11:42:02Z] <topranks> cr3-eqsin, begin traffic drain to reset PIC and upgrade JunOS T429386

Icinga downtime and Alertmanager silence (ID=99c0df32-80dc-453a-920f-5d3957387031) set by cmooney@cumin1003 for 1:00:00 on 3 host(s) and their services with reason: upgrade JunOS cr3-eqsin

lvs[5004-5006].eqsin.wmnet

Icinga downtime and Alertmanager silence (ID=8f5cefde-0060-47c2-8575-70bfa00d21f7) set by cmooney@cumin1003 for 1:00:00 on 5 host(s) and their services with reason: upgrade JunOS cr3-eqsin

cr1-codfw,cr[2-3]-eqsin,cr3-eqsin IPv6,cr3-eqsin.mgmt

Mentioned in SAL (#wikimedia-operations) [2026-07-07T12:58:32Z] <topranks> load updated JunOS on cr3-eqsin T429386

Upgraded cr3-eqsin also as PIC needed bouncing. And created some instructions to do the traffic drain without 'graceful-shutdown' P94750.

ayounsi subscribed.

Closing this task as all the problematic 23.4R2-S7 have been upgraded. The remaining ones will go directly to S8