Page MenuHomePhabricator

Core router upgrades 2026 #2
Open, MediumPublic

Description

Under T416444 we have been upgrading our Juniper devices, including core routers, to address some mid-level CVEs.

Unfortunately as part of this process we hit on a bug which can cause rpd to crash on JunOS when the BGP graceful-shutdown feature is used (see T429386). Juniper have informed us this is a known bug which is only fixed in the very latest point release 23.4R2-S8.

The good thing is there is no massive urgency to upgrade to address this issue. It only affects manual operations that we might carry out, and doesn't represent a security or operational risk otherwise (i.e. without one of us issuing the graceful-shutdown command).

The bad thing is we use that command when draining traffic on routers to upgrade them, and Juniper advise it is present in all previous JunOS verions (though we've only hit it on 23.4R2-S7).

Creating this task to track:

  • The remaining upgrades already planned, which we will now skip to 23.4R2-S8
  • The core routers we've already done and put on 23.4R2-S7, which we now need to revisit.

Event Timeline

cmooney triaged this task as Medium priority.

We don't necessarily have to follow this process, but a rough outline of how I drained the recent routers I upgraded without using graceful-shutdown is below

1
2# Drain active transport with OSPF Cost
3
4# Pre-pend routes we announce to peering and transit
5#* Also delete the 'export' policy for any specific peers completely:
6#* show configuration protocols bgp | display set | match export | match neigh
7# any that come back - delete protocols bgp group <group> neighbor <neighbor> export
8
9delete protocols bgp group Transit4 export BGP_accept
10delete protocols bgp group Transit6 export BGP_accept
11delete protocols bgp group IX4 export BGP_accept
12delete protocols bgp group IX6 export BGP_accept
13delete protocols bgp group Private-Peer4 export BGP_accept
14delete protocols bgp group Private-Peer6 export BGP_accept
15
16set protocols bgp group Transit4 export BGP_wmf_prepend
17set protocols bgp group Transit6 export BGP_wmf_prepend
18set protocols bgp group IX4 export BGP_wmf_prepend
19set protocols bgp group IX6 export BGP_wmf_prepend
20set protocols bgp group Private-Peer4 export BGP_wmf_prepend
21set protocols bgp group Private-Peer6 export BGP_wmf_prepend
22
23# Pre-pend routes we announce to our Switches on site
24
25set policy-options policy-statement Switch_out term direct then as-path-prepend "14907 14907 14907"
26set policy-options policy-statement Switch_out term ospf then as-path-prepend "14907 14907 14907"
27set policy-options policy-statement Switch_out term default4 then as-path-prepend "14907 14907 14907"
28set policy-options policy-statement Switch_out term default6 then as-path-prepend "14907 14907 14907"
29
30
31# Set local-pref to 0 and MED high for all routes we are learning:
32#* Also delete any peer-specific import policies in affected groups
33#* show configuration protocols bgp | display set | match import | match neigh
34
35set policy-options policy-statement PREF0 then metric 10000
36set policy-options policy-statement PREF0 then local-preference 0
37set policy-options policy-statement PREF0 then accept
38
39delete protocols bgp group Transit4 import
40delete protocols bgp group Transit6 import
41delete protocols bgp group IX4 import
42delete protocols bgp group IX6 import
43delete protocols bgp group Private-Peer4 import
44delete protocols bgp group Private-Peer6 import
45delete protocols bgp group Switch import
46
47set protocols bgp group Transit4 import BGP_sanitize_in
48set protocols bgp group Transit4 import PREF0
49
50set protocols bgp group Transit6 import BGP_sanitize_in
51set protocols bgp group Transit6 import PREF0
52
53set protocols bgp group IX4 import BGP_sanitize_in
54set protocols bgp group IX4 import PREF0
55
56set protocols bgp group IX6 import BGP_sanitize_in
57set protocols bgp group IX6 import PREF0
58
59set protocols bgp group Private-Peer4 import BGP_sanitize_in
60set protocols bgp group Private-Peer4 import PREF0
61
62set protocols bgp group Private-Peer6 import BGP_sanitize_in
63set protocols bgp group Private-Peer6 import PREF0
64
65set protocols bgp group Switch import PREF0
66
67# Change VRRP priority if appropriate
68# show configuration | display set | match vrrp | match priority | except cost | match 110
69# replace '110' with '50' and commit, check router is backup for all groups in "show vrrp summary"
70
71
72# After 10-15 mins withdraw all routes to external peering / transit
73
74delete protocols bgp group Transit4 export
75delete protocols bgp group Transit6 export
76delete protocols bgp group IX4 export
77delete protocols bgp group IX6 export
78delete protocols bgp group Private-Peer4 export
79delete protocols bgp group Private-Peer6 export
80
81set protocols bgp group Transit4 export NONE
82set protocols bgp group Transit6 export NONE
83set protocols bgp group IX4 export NONE
84set protocols bgp group IX6 export NONE
85set protocols bgp group Private-Peer4 export NONE
86set protocols bgp group Private-Peer6 export NONE
87

ayounsi claimed this task.
ayounsi subscribed.
This comment was removed by ayounsi.
ayounsi removed ayounsi as the assignee of this task.