Page MenuHomePhabricator

Announce internal/core routes from CRs to L3 switches
Closed, ResolvedPublic

Description

Background

We currently announce a default route (both v4 and v6) to layer-3 switches from our core/border routers.

This makes sense, the switch devices cannot hold a full internet routing table, and also only have uplinks to the CRs where they send all traffic so there is no benefit to them having all the routes.

Issue

There are some potential issues with this, however. The default route is synthetically generated by the CRs and sent to the switches. Specifically the CRs will announce the default whether or not they have any other connectivity working. An example was during the esams build when cr1-esams was initially connected to the switches. It had no external connectivity or link to the other CR in the same POP. So it could not reach any destinations at all. But still it sent the default to the layer-3 switches.

As a result when it came up the switches started sending traffic to it, which it just dropped. This blackholed the site for remote users, despite cr2-esams still being connected (announcing a default too). While this scenario is quite unusual, there are probably other edge-case scenarios where the generated default could result in dropped traffic or sub-optimal routing. There is no pressure or urgency to address really, but we might want to consider.

Solution

I think it would make sense if we send, in addition to the defaults, all the internal Wikimedia routes to the switches. These are not synthetically generated, but distributed in our core with BGP/OSPF. As such they are a real representation of network reachability to the destinations.

In the scenario which affected traffic during the esams rollout what this would mean is that when cr1-esams came up and sent the default route that would still have disrupted traffic from the esams switches going to the internet. But for our internal traffic (crucially our own reachability to devices etc.) the switches would have chosen the path via cr2-esams. That's because cr2-esams would be the only one sending the more-specific internal routers, trumping the default coming from the router with no outbound connectivity.

Example

I trialled this while esams was depooled. Firstly defined a new policy on cr2-esams as follows, applied to the 'Switch' BGP group:

cmooney@cr2-esams> show configuration policy-options policy-statement Switch_out | display set 
set policy-options policy-statement Switch_out term from_ospf4 from protocol ospf
set policy-options policy-statement Switch_out term from_ospf4 then accept
set policy-options policy-statement Switch_out term from_osfp6 from protocol ospf3
set policy-options policy-statement Switch_out term from_osfp6 then accept
set policy-options policy-statement Switch_out term from_bgp from as-path core_and_local_LVS
set policy-options policy-statement Switch_out term from_bgp from as-path anycast
set policy-options policy-statement Switch_out term from_bgp from as-path from_remote_confed
set policy-options policy-statement Switch_out term from_bgp then accept
set policy-options policy-statement Switch_out term default4 from route-filter 0.0.0.0/0 exact
set policy-options policy-statement Switch_out term default4 then accept
set policy-options policy-statement Switch_out term default6 from route-filter ::/0 exact
set policy-options policy-statement Switch_out term default6 then accept
set policy-options policy-statement Switch_out then reject

On the switch side I removed the import filter (we can instead use a tailored one if we decide to deploy), and added a max-prefix statement for safety:

cmooney@asw1-by27-esams> show configuration protocols bgp group Production | display set            
set protocols bgp group Production type external
set protocols bgp group Production export BGP_Production_out
set protocols bgp group Production peer-as 14907
set protocols bgp group Production multipath
set protocols bgp group Production neighbor 185.15.59.150 description cr2-esams
set protocols bgp group Production neighbor 185.15.59.150 family inet unicast prefix-limit maximum 5000
set protocols bgp group Production neighbor 2a02:ec80:300:fe05::1 description cr1-esams
set protocols bgp group Production neighbor 2a02:ec80:300:fe05::1 family inet6 unicast prefix-limit maximum 5000
set protocols bgp group Production neighbor 2a02:ec80:300:fe08::1 description cr2-esams
set protocols bgp group Production neighbor 2a02:ec80:300:fe08::1 family inet6 unicast prefix-limit maximum 5000
set protocols bgp group Production neighbor 185.15.59.154 description cr1-esams
set protocols bgp group Production neighbor 185.15.59.154 family inet unicast prefix-limit maximum 5000

This resulted in the switches learning routes as follows from cr2:

cmooney@asw1-by27-esams> show route receive-protocol bgp 185.15.59.150 terse table inet.0           

Warning: License key missing; requires 'bgp' license


inet.0: 531 destinations, 534 routes (530 active, 0 holddown, 2 hidden)
  Prefix		  Nexthop	       MED     Lclpref    AS path
* 0.0.0.0/0               185.15.59.150                           14907 I
* 4.15.72.112/30          185.15.59.150        1957               14907 I
* 4.16.71.244/30          185.15.59.150        1820               14907 I
* 4.53.96.36/30           185.15.59.150        1800               14907 I
* 10.2.1.0/24             185.15.59.150        0                  14907 I
* 10.2.1.1/32             185.15.59.150                           14907 64600 I
* 10.2.1.5/32             185.15.59.150                           14907 64600 I
* 10.2.1.8/32             185.15.59.150                           14907 64600 I
* 10.2.1.10/32            185.15.59.150                           14907 64600 I
* 10.2.1.11/32            185.15.59.150                           14907 64600 I
* 10.2.1.12/32            185.15.59.150                           14907 64600 I
* 10.2.1.13/32            185.15.59.150                           14907 64600 I
* 10.2.1.14/32            185.15.59.150                           14907 64600 I
* 10.2.1.16/32            185.15.59.150                           14907 64600 I
* 10.2.1.17/32            185.15.59.150                           14907 64600 I
* 10.2.1.18/32            185.15.59.150                           14907 64600 I
* 10.2.1.19/32            185.15.59.150                           14907 64600 I
* 10.2.1.20/32            185.15.59.150                           14907 64600 I
* 10.2.1.21/32            185.15.59.150                           14907 64600 I
* 10.2.1.22/32            185.15.59.150                           14907 64600 I
* 10.2.1.23/32            185.15.59.150                           14907 64600 I
* 10.2.1.24/32            185.15.59.150                           14907 64600 I
* 10.2.1.25/32            185.15.59.150                           14907 64600 I
* 10.2.1.26/32            185.15.59.150                           14907 64600 I
* 10.2.1.27/32            185.15.59.150                           14907 64600 I
* 10.2.1.28/32            185.15.59.150                           14907 64600 I
* 10.2.1.29/32            185.15.59.150                           14907 64600 I
* 10.2.1.30/32            185.15.59.150                           14907 64600 I
* 10.2.1.31/32            185.15.59.150                           14907 64600 I
* 10.2.1.32/32            185.15.59.150                           14907 64600 I
* 10.2.1.34/32            185.15.59.150                           14907 64600 I
* 10.2.1.35/32            185.15.59.150                           14907 64600 I
* 10.2.1.37/32            185.15.59.150                           14907 64600 I
* 10.2.1.39/32            185.15.59.150                           14907 64600 I
* 10.2.1.41/32            185.15.59.150                           14907 64600 I
* 10.2.1.42/32            185.15.59.150                           14907 64600 I
* 10.2.1.43/32            185.15.59.150                           14907 64600 I
* 10.2.1.44/32            185.15.59.150                           14907 64600 I
* 10.2.1.45/32            185.15.59.150                           14907 64600 I
* 10.2.1.46/32            185.15.59.150                           14907 64600 I
* 10.2.1.47/32            185.15.59.150                           14907 64600 I
* 10.2.1.48/32            185.15.59.150                           14907 64600 I
* 10.2.1.49/32            185.15.59.150                           14907 64600 I
* 10.2.1.50/32            185.15.59.150                           14907 64600 I
* 10.2.1.51/32            185.15.59.150                           14907 64600 I
* 10.2.1.52/32            185.15.59.150                           14907 64600 I
* 10.2.1.53/32            185.15.59.150                           14907 64600 I
* 10.2.1.54/32            185.15.59.150                           14907 64600 I
* 10.2.1.55/32            185.15.59.150                           14907 64600 I
* 10.2.1.56/32            185.15.59.150                           14907 64600 I
* 10.2.1.57/32            185.15.59.150                           14907 64600 I
* 10.2.1.58/32            185.15.59.150                           14907 64600 I
* 10.2.1.59/32            185.15.59.150                           14907 64600 I
* 10.2.1.60/32            185.15.59.150                           14907 64600 I
* 10.2.1.61/32            185.15.59.150                           14907 64600 I
* 10.2.1.62/32            185.15.59.150                           14907 64600 I
* 10.2.1.63/32            185.15.59.150                           14907 64600 I
* 10.2.1.64/32            185.15.59.150                           14907 64600 I
* 10.2.1.65/32            185.15.59.150                           14907 64600 I
* 10.2.1.66/32            185.15.59.150                           14907 64600 I
* 10.2.1.67/32            185.15.59.150                           14907 64600 I
* 10.2.1.69/32            185.15.59.150                           14907 64600 I
* 10.2.1.70/32            185.15.59.150                           14907 64600 I
* 10.2.1.72/32            185.15.59.150                           14907 64600 I
* 10.2.1.75/32            185.15.59.150                           14907 64600 I
* 10.2.1.76/32            185.15.59.150                           14907 64600 I
* 10.2.1.77/32            185.15.59.150                           14907 64600 I
* 10.2.1.79/32            185.15.59.150                           14907 64600 I
* 10.2.1.80/32            185.15.59.150                           14907 64600 I
* 10.2.1.81/32            185.15.59.150                           14907 64600 I
* 10.2.1.82/32            185.15.59.150                           14907 64600 I
* 10.2.1.83/32            185.15.59.150                           14907 64600 I
* 10.2.1.84/32            185.15.59.150                           14907 64600 I
* 10.2.1.85/32            185.15.59.150                           14907 64600 I
* 10.2.1.86/32            185.15.59.150                           14907 64600 I
* 10.2.2.0/24             185.15.59.150        0                  14907 I
* 10.2.2.1/32             185.15.59.150                           14907 64600 I
* 10.2.2.5/32             185.15.59.150                           14907 64600 I
* 10.2.2.8/32             185.15.59.150                           14907 64600 I
* 10.2.2.10/32            185.15.59.150                           14907 64600 I
* 10.2.2.11/32            185.15.59.150                           14907 64600 I
* 10.2.2.12/32            185.15.59.150                           14907 64600 I
* 10.2.2.13/32            185.15.59.150                           14907 64600 I
* 10.2.2.14/32            185.15.59.150                           14907 64600 I
* 10.2.2.16/32            185.15.59.150                           14907 64600 I
* 10.2.2.17/32            185.15.59.150                           14907 64600 I
* 10.2.2.18/32            185.15.59.150                           14907 64600 I
* 10.2.2.19/32            185.15.59.150                           14907 64600 I
* 10.2.2.20/32            185.15.59.150                           14907 64600 I
* 10.2.2.21/32            185.15.59.150                           14907 64600 I
* 10.2.2.22/32            185.15.59.150                           14907 64600 I
* 10.2.2.23/32            185.15.59.150                           14907 64600 I
* 10.2.2.24/32            185.15.59.150                           14907 64600 I
* 10.2.2.25/32            185.15.59.150                           14907 64600 I
* 10.2.2.26/32            185.15.59.150                           14907 64600 I
* 10.2.2.27/32            185.15.59.150                           14907 64600 I
* 10.2.2.28/32            185.15.59.150                           14907 64600 I
* 10.2.2.29/32            185.15.59.150                           14907 64600 I
* 10.2.2.30/32            185.15.59.150                           14907 64600 I
* 10.2.2.31/32            185.15.59.150                           14907 64600 I
* 10.2.2.32/32            185.15.59.150                           14907 64600 I
* 10.2.2.34/32            185.15.59.150                           14907 64600 I
* 10.2.2.35/32            185.15.59.150                           14907 64600 I
* 10.2.2.37/32            185.15.59.150                           14907 64600 I
* 10.2.2.38/32            185.15.59.150                           14907 64600 I
* 10.2.2.39/32            185.15.59.150                           14907 64600 I
* 10.2.2.40/32            185.15.59.150                           14907 64600 I
* 10.2.2.41/32            185.15.59.150                           14907 64600 I
* 10.2.2.42/32            185.15.59.150                           14907 64600 I
* 10.2.2.43/32            185.15.59.150                           14907 64600 I
* 10.2.2.44/32            185.15.59.150                           14907 64600 I
* 10.2.2.45/32            185.15.59.150                           14907 64600 I
* 10.2.2.46/32            185.15.59.150                           14907 64600 I
* 10.2.2.47/32            185.15.59.150                           14907 64600 I
* 10.2.2.48/32            185.15.59.150                           14907 64600 I
* 10.2.2.49/32            185.15.59.150                           14907 64600 I
* 10.2.2.50/32            185.15.59.150                           14907 64600 I
* 10.2.2.51/32            185.15.59.150                           14907 64600 I
* 10.2.2.52/32            185.15.59.150                           14907 64600 I
* 10.2.2.53/32            185.15.59.150                           14907 64600 I
* 10.2.2.54/32            185.15.59.150                           14907 64600 I
* 10.2.2.55/32            185.15.59.150                           14907 64600 I
* 10.2.2.56/32            185.15.59.150                           14907 64600 I
* 10.2.2.57/32            185.15.59.150                           14907 64600 I
* 10.2.2.59/32            185.15.59.150                           14907 64600 I
* 10.2.2.60/32            185.15.59.150                           14907 64600 I
* 10.2.2.61/32            185.15.59.150                           14907 64600 I
* 10.2.2.62/32            185.15.59.150                           14907 64600 I
* 10.2.2.63/32            185.15.59.150                           14907 64600 I
* 10.2.2.64/32            185.15.59.150                           14907 64600 I
* 10.2.2.65/32            185.15.59.150                           14907 64600 I
* 10.2.2.66/32            185.15.59.150                           14907 64600 I
* 10.2.2.67/32            185.15.59.150                           14907 64600 I
* 10.2.2.69/32            185.15.59.150                           14907 64600 I
* 10.2.2.70/32            185.15.59.150                           14907 64600 I
* 10.2.2.71/32            185.15.59.150                           14907 64600 I
* 10.2.2.73/32            185.15.59.150                           14907 64600 I
* 10.2.2.74/32            185.15.59.150                           14907 64600 I
* 10.2.2.75/32            185.15.59.150                           14907 64600 I
* 10.2.2.76/32            185.15.59.150                           14907 64600 I
* 10.2.2.77/32            185.15.59.150                           14907 64600 I
* 10.2.2.79/32            185.15.59.150                           14907 64600 I
* 10.2.2.80/32            185.15.59.150                           14907 64600 I
* 10.2.2.81/32            185.15.59.150                           14907 64600 I
* 10.2.2.82/32            185.15.59.150                           14907 64600 I
* 10.2.2.84/32            185.15.59.150                           14907 64600 I
* 10.2.2.85/32            185.15.59.150                           14907 64600 I
* 10.2.2.86/32            185.15.59.150                           14907 64600 I
* 10.3.0.0/30             185.15.59.150        0                  14907 I
* 10.3.0.4/32             185.15.59.150                           14907 64605 64605 I
* 10.64.0.0/22            185.15.59.150        1822               14907 I
* 10.64.4.0/24            185.15.59.150        1822               14907 I
* 10.64.5.0/24            185.15.59.150        1822               14907 I
* 10.64.16.0/22           185.15.59.150        1822               14907 I
* 10.64.20.0/24           185.15.59.150                           14907 64710 I
* 10.64.21.0/24           185.15.59.150        1822               14907 I
* 10.64.32.0/22           185.15.59.150        1822               14907 I
* 10.64.36.0/24           185.15.59.150        1822               14907 I
* 10.64.37.0/24           185.15.59.150        1822               14907 I
* 10.64.40.0/27           185.15.59.150                           14907 64700 I
* 10.64.40.32/27          185.15.59.150                           14907 64700 I
* 10.64.40.64/27          185.15.59.150                           14907 64700 I
* 10.64.40.96/27          185.15.59.150                           14907 64700 I
* 10.64.40.160/27         185.15.59.150                           14907 64700 I
* 10.64.40.192/26         185.15.59.150                           14907 64700 I
* 10.64.48.0/22           185.15.59.150        1822               14907 I
* 10.64.53.0/24           185.15.59.150        1822               14907 I
* 10.64.75.0/26           185.15.59.150                           14907 64603 I
* 10.64.75.64/26          185.15.59.150                           14907 64603 I
* 10.64.75.128/26         185.15.59.150                           14907 64603 I
* 10.64.75.192/26         185.15.59.150                           14907 64603 I
* 10.64.130.0/24          185.15.59.150                           14907 64810 I
* 10.64.131.0/24          185.15.59.150                           14907 64810 I
* 10.64.132.0/24          185.15.59.150                           14907 64810 I
* 10.64.134.0/24          185.15.59.150                           14907 64810 I
* 10.64.135.0/24          185.15.59.150                           14907 64810 I
* 10.64.136.0/24          185.15.59.150                           14907 64810 I
* 10.64.138.0/24          185.15.59.150                           14907 64810 I
* 10.64.139.0/24          185.15.59.150                           14907 64810 I
* 10.64.140.0/24          185.15.59.150                           14907 64810 I
* 10.64.142.0/24          185.15.59.150                           14907 64810 I
* 10.64.143.0/24          185.15.59.150                           14907 64810 I
* 10.64.144.0/24          185.15.59.150                           14907 64810 I
* 10.64.146.1/32          185.15.59.150                           14907 64810 I
* 10.64.146.2/32          185.15.59.150                           14907 64810 I
* 10.64.146.3/32          185.15.59.150                           14907 64810 I
* 10.64.146.4/32          185.15.59.150                           14907 64810 I
* 10.64.146.5/32          185.15.59.150                           14907 64810 I
* 10.64.146.7/32          185.15.59.150                           14907 64810 I
* 10.64.146.8/32          185.15.59.150                           14907 64810 I
* 10.64.146.9/32          185.15.59.150                           14907 64810 I
* 10.64.146.251/32        185.15.59.150                           14907 64710 I
* 10.64.146.252/32        185.15.59.150                           14907 64710 I
* 10.64.146.253/32        185.15.59.150                           14907 64710 I
* 10.64.146.254/32        185.15.59.150                           14907 64710 4264710003 I
* 10.64.146.255/32        185.15.59.150                           14907 64710 4264710004 I
* 10.64.147.14/31         185.15.59.150        1823               14907 I
* 10.64.147.16/31         185.15.59.150        1820               14907 I
* 10.64.148.0/24          185.15.59.150                           14907 64710 4264710003 I
* 10.64.149.0/24          185.15.59.150                           14907 64710 4264710004 I
* 10.64.150.0/24          185.15.59.150                           14907 64710 I
* 10.64.151.0/24          185.15.59.150                           14907 64710 I
* 10.65.0.0/16            185.15.59.150        21920              14907 I
* 10.66.0.0/30            185.15.59.150        1820               14907 I
* 10.66.0.4/30            185.15.59.150        1823               14907 I
* 10.66.0.8/31            185.15.59.150        1820               14907 I
* 10.66.0.10/31           185.15.59.150        1823               14907 I
* 10.67.16.128/26         185.15.59.150                           14907 64810 64606 I
* 10.67.17.192/26         185.15.59.150                           14907 64606 I
* 10.67.21.64/26          185.15.59.150                           14907 64606 I
* 10.67.21.128/26         185.15.59.150                           14907 64810 64606 I
* 10.67.22.64/26          185.15.59.150                           14907 64606 I
* 10.67.23.0/26           185.15.59.150                           14907 64606 I
* 10.67.23.128/26         185.15.59.150                           14907 64810 64606 I
* 10.67.23.192/26         185.15.59.150                           14907 64810 64606 I
* 10.67.24.64/26          185.15.59.150                           14907 64609 I
* 10.67.25.0/26           185.15.59.150                           14907 64810 64609 I
* 10.67.26.64/26          185.15.59.150                           14907 64810 64609 I
* 10.67.27.192/26         185.15.59.150                           14907 64609 I
* 10.67.28.0/26           185.15.59.150                           14907 64810 64609 I
* 10.67.29.192/26         185.15.59.150                           14907 64609 I
* 10.67.30.128/26         185.15.59.150                           14907 64609 I
* 10.67.30.192/26         185.15.59.150                           14907 64810 64609 I
* 10.67.83.192/26         185.15.59.150                           14907 64610 I
* 10.67.87.0/26           185.15.59.150                           14907 64610 I
* 10.67.130.128/26        185.15.59.150                           14907 64601 I
* 10.67.133.128/26        185.15.59.150                           14907 64601 I
* 10.67.135.0/26          185.15.59.150                           14907 64601 I
* 10.67.140.128/26        185.15.59.150                           14907 64601 I
* 10.67.142.128/26        185.15.59.150                           14907 64601 I
* 10.67.147.64/26         185.15.59.150                           14907 64601 I
* 10.67.149.192/26        185.15.59.150                           14907 64601 I
* 10.67.151.192/26        185.15.59.150                           14907 64601 I
* 10.67.155.192/26        185.15.59.150                           14907 64601 I
* 10.67.156.128/26        185.15.59.150                           14907 64601 I
* 10.67.157.128/26        185.15.59.150                           14907 64601 I
* 10.67.158.128/26        185.15.59.150                           14907 64601 I
* 10.67.161.0/26          185.15.59.150                           14907 64601 I
* 10.67.166.64/26         185.15.59.150                           14907 64601 I
* 10.67.166.128/26        185.15.59.150                           14907 64810 64601 I
* 10.67.168.0/26          185.15.59.150                           14907 64601 I
* 10.67.169.192/26        185.15.59.150                           14907 64810 64601 I
* 10.67.172.0/26          185.15.59.150                           14907 64601 I
* 10.67.173.128/26        185.15.59.150                           14907 64601 I
* 10.67.175.64/26         185.15.59.150                           14907 64601 I
* 10.67.178.64/26         185.15.59.150                           14907 64601 I
* 10.67.180.128/26        185.15.59.150                           14907 64601 I
* 10.67.180.192/26        185.15.59.150                           14907 64601 I
* 10.67.185.0/26          185.15.59.150                           14907 64601 I
* 10.128.0.0/24           185.15.59.150        1957               14907 I
* 10.128.128.0/17         185.15.59.150        22055              14907 I
* 10.132.0.0/24           185.15.59.150        3567               14907 I
* 10.132.4.0/30           185.15.59.150        3567               14907 I
* 10.132.4.4/30           185.15.59.150        3565               14907 I
* 10.132.128.0/17         185.15.59.150        23665              14907 I
* 10.136.0.0/24           185.15.59.150                           14907 4265006001 I
* 10.136.1.0/24           185.15.59.150                           14907 4265006002 I
* 10.136.127.2/31         185.15.59.150                           14907 4265006001 I
* 10.136.127.4/31         185.15.59.150                           14907 4265006002 I
* 10.136.127.6/31         185.15.59.150        2700               14907 I
* 10.136.127.8/31         185.15.59.150        2702               14907 I
* 10.136.128.0/17         185.15.59.150                           14907 4265006001 65000 I
* 10.192.0.0/22           185.15.59.150        1562               14907 I
* 10.192.16.0/22          185.15.59.150        1562               14907 I
* 10.192.20.0/24          185.15.59.150                           14907 64711 I
* 10.192.21.0/24          185.15.59.150        1562               14907 I
* 10.192.32.0/22          185.15.59.150        1562               14907 I
* 10.192.48.0/22          185.15.59.150        1562               14907 I
* 10.192.75.0/26          185.15.59.150                           14907 64604 I
* 10.192.75.64/26         185.15.59.150                           14907 64604 I
* 10.192.75.128/26        185.15.59.150                           14907 64604 I
* 10.192.75.192/26        185.15.59.150                           14907 64604 I
* 10.192.254.0/31         185.15.59.150        1565               14907 I
* 10.192.254.2/31         185.15.59.150        1560               14907 I
* 10.192.255.255/32       185.15.59.150                           14907 64711 I
* 10.193.0.0/16           185.15.59.150        21660              14907 I
* 10.194.16.64/26         185.15.59.150                           14907 64607 I
* 10.194.18.0/26          185.15.59.150                           14907 64607 I
* 10.194.18.192/26        185.15.59.150                           14907 64607 I
* 10.194.19.64/26         185.15.59.150                           14907 64607 I
* 10.194.20.128/26        185.15.59.150                           14907 64607 I
* 10.194.20.192/26        185.15.59.150                           14907 64607 I
* 10.194.22.192/26        185.15.59.150                           14907 64607 I
* 10.194.23.128/26        185.15.59.150                           14907 64607 I
* 10.194.61.128/26        185.15.59.150                           14907 64608 I
* 10.194.61.192/26        185.15.59.150                           14907 64608 I
* 10.194.134.0/26         185.15.59.150                           14907 64602 I
* 10.194.134.192/26       185.15.59.150                           14907 64602 I
* 10.194.135.64/26        185.15.59.150                           14907 64602 I
* 10.194.137.128/26       185.15.59.150                           14907 64602 I
* 10.194.139.64/26        185.15.59.150                           14907 64602 I
* 10.194.147.128/26       185.15.59.150                           14907 64602 I
* 10.194.149.128/26       185.15.59.150                           14907 64602 I
* 10.194.150.0/26         185.15.59.150                           14907 64602 I
* 10.194.153.128/26       185.15.59.150                           14907 64602 I
* 10.194.157.64/26        185.15.59.150                           14907 64602 I
* 10.194.161.128/26       185.15.59.150                           14907 64602 I
* 10.194.167.0/26         185.15.59.150                           14907 64602 I
* 10.194.171.192/26       185.15.59.150                           14907 64602 I
* 10.194.173.64/26        185.15.59.150                           14907 64602 I
* 10.194.173.192/26       185.15.59.150                           14907 64602 I
* 10.194.179.128/26       185.15.59.150                           14907 64602 I
* 10.194.181.0/26         185.15.59.150                           14907 64602 I
* 10.194.183.128/26       185.15.59.150                           14907 64602 I
* 10.194.184.192/26       185.15.59.150                           14907 64602 I
* 10.194.185.128/26       185.15.59.150                           14907 64602 I
* 10.194.187.0/26         185.15.59.150                           14907 64602 I
* 10.194.191.128/26       185.15.59.150                           14907 64602 I
* 10.195.0.0/27           185.15.59.150                           14907 64701 I
* 10.195.0.32/27          185.15.59.150                           14907 64701 I
* 10.195.0.64/28          185.15.59.150                           14907 64701 I
* 10.195.0.80/29          185.15.59.150                           14907 64701 I
* 10.195.0.96/27          185.15.59.150                           14907 64701 I
* 10.195.0.128/29         185.15.59.150                           14907 64701 I
* 10.195.0.248/31         185.15.59.150                           14907 64701 I
* 27.111.228.0/22         185.15.59.150        3565               14907 I
* 37.49.232.0/24          185.15.59.150        2702               14907 I
* 62.115.147.132/31       185.15.59.150        2700               14907 I
* 62.115.148.76/31        185.15.59.150        3565               14907 I
* 62.115.179.162/31       185.15.59.150        3                  14907 I
* 64.156.73.168/30        185.15.59.150        1560               14907 I
* 80.231.165.54/31        185.15.59.150        2700               14907 I
* 80.239.132.224/30       185.15.59.150        1823               14907 I
* 80.239.192.64/30        185.15.59.150        1957               14907 I
* 80.239.192.100/30       185.15.59.150        1565               14907 I
* 80.249.208.0/21         185.15.59.150        3                  14907 I
* 81.93.217.52/31         185.15.59.150        2702               14907 I
* 91.198.174.224/31       185.15.59.150        5003               14907 I
* 91.198.174.248/31       185.15.59.150        5003               14907 I
* 103.16.102.0/23         185.15.59.150        3567               14907 I
* 103.102.166.0/24        185.15.59.150        0                  14907 I
* 103.102.166.0/28        185.15.59.150        3567               14907 I
* 103.102.166.16/28       185.15.59.150        3567               14907 I
* 103.102.166.128/32      185.15.59.150        23565              14907 I
* 103.102.166.130/32      185.15.59.150        3567               14907 I
* 103.102.166.131/32      185.15.59.150        3565               14907 I
* 103.102.166.132/31      185.15.59.150        23565              14907 I
* 103.102.166.134/31      185.15.59.150        3567               14907 I
* 103.102.166.138/31      185.15.59.150        3565               14907 I
* 103.102.166.140/31      185.15.59.150        3567               14907 I
* 103.102.166.142/31      185.15.59.150        23567              14907 I
* 103.102.166.144/31      185.15.59.150        3565               14907 I
* 103.102.166.146/31      185.15.59.150        4550               14907 I
* 103.102.166.224/27      185.15.59.150        0                  14907 I
* 103.102.166.224/28      185.15.59.150        0                  14907 I
* 103.102.166.224/32      185.15.59.150                           14907 64600 I
* 103.102.166.225/32      185.15.59.150                           14907 64600 I
* 103.102.166.226/32      185.15.59.150                           14907 64600 I
* 103.102.166.227/32      185.15.59.150                           14907 64600 I
* 103.102.166.240/28      185.15.59.150        0                  14907 I
* 103.102.166.240/32      185.15.59.150                           14907 64600 I
* 103.102.166.241/32      185.15.59.150                           14907 64600 I
* 103.231.152.0/22        185.15.59.150        3567               14907 I
* 116.51.26.208/30        185.15.59.150        3567               14907 I
* 116.51.26.210/32        185.15.59.150        0                  14907 I
* 128.242.179.180/30      185.15.59.150        1550               14907 I
* 128.242.179.182/32      185.15.59.150        0                  14907 I
* 129.250.204.4/30        185.15.59.150        1955               14907 I
* 129.250.204.6/32        185.15.59.150        0                  14907 I
* 157.240.76.224/31       185.15.59.150        1550               14907 I
* 157.240.78.74/31        185.15.59.150        1820               14907 I
* 162.159.64.7/32         185.15.59.150        0                  14907 I
* 172.16.0.0/21           185.15.59.150                           14907 64710 I
* 172.16.128.0/21         185.15.59.150                           14907 64711 I
* 180.87.164.60/30        185.15.59.150        3567               14907 I
* 185.1.47.0/24           185.15.59.150        2700               14907 I
* 185.15.56.0/24          185.15.59.150                           14907 64710 ?
* 185.15.57.0/24          185.15.59.150        0                  14907 I
* 185.15.57.0/27          185.15.59.150                           14907 64711 ?
* 185.15.58.0/24          185.15.59.150        0                  14907 I
* 185.15.58.0/27          185.15.59.150                           14907 4265006001 I
* 185.15.58.32/27         185.15.59.150                           14907 4265006002 I
* 185.15.58.128/32        185.15.59.150        2700               14907 I
* 185.15.58.129/32        185.15.59.150        2702               14907 I
* 185.15.58.130/32        185.15.59.150                           14907 4265006001 65000 I
* 185.15.58.131/32        185.15.59.150                           14907 4265006001 I
* 185.15.58.132/32        185.15.59.150                           14907 4265006002 I
* 185.15.58.136/31        185.15.59.150        2702               14907 I
* 185.15.58.138/31        185.15.59.150        2700               14907 I
* 185.15.58.140/31        185.15.59.150        2702               14907 I
* 185.15.58.142/31        185.15.59.150        2700               14907 I
* 185.15.58.144/31        185.15.59.150        2702               14907 I
* 185.15.58.146/31        185.15.59.150        2788               14907 I
* 185.15.58.148/31        185.15.59.150        2700               14907 I
* 185.15.58.224/27        185.15.59.150        0                  14907 I
* 185.15.58.224/32        185.15.59.150                           14907 4265006001 64600 I
* 185.15.58.225/32        185.15.59.150                           14907 4265006001 64600 I
* 185.15.58.226/32        185.15.59.150                           14907 4265006001 64600 I
* 185.15.58.227/32        185.15.59.150                           14907 4265006001 64600 I
* 185.15.58.240/32        185.15.59.150                           14907 4265006002 64600 I
* 185.15.58.241/32        185.15.59.150                           14907 4265006002 64600 I
* 185.15.59.128/32        185.15.59.150        3                  14907 I
* 185.15.59.146/31        185.15.59.150        5003               14907 I
* 185.15.59.148/31        185.15.59.150        5003               14907 I
  185.15.59.154/31        185.15.59.150        3                  14907 I
* 185.15.59.156/31        185.15.59.150        3                  14907 I
* 185.71.138.0/24         185.15.59.150        0                  14907 I
* 185.71.138.138/32       185.15.59.150                           14907 64605 64605 I
* 185.71.138.139/32       185.15.59.150                           14907 64605 64605 I
* 185.71.138.140/32       185.15.59.150                           14907 64605 64605 I
* 185.71.138.141/32       185.15.59.150                           14907 64605 64605 I
* 185.71.138.142/32       185.15.59.150                           14907 64605 64605 I
* 192.80.17.184/30        185.15.59.150        1820               14907 I
* 192.80.17.188/30        185.15.59.150        1550               14907 I
* 192.80.17.190/32        185.15.59.150        0                  14907 I
* 192.80.17.192/30        185.15.59.150        1955               14907 I
* 192.80.17.194/32        185.15.59.150        0                  14907 I
* 192.80.17.196/30        185.15.59.150        1800               14907 I
* 193.251.154.144/31      185.15.59.150        2702               14907 I
* 194.62.128.0/24         185.15.59.150        3                  14907 I
* 198.35.26.0/24          185.15.59.150        0                  14907 I
* 198.35.26.0/28          185.15.59.150        1957               14907 I
* 198.35.26.96/27         185.15.59.150        0                  14907 I
* 198.35.26.96/28         185.15.59.150        0                  14907 I
* 198.35.26.96/32         185.15.59.150                           14907 64600 I
* 198.35.26.97/32         185.15.59.150                           14907 64600 I
* 198.35.26.98/32         185.15.59.150                           14907 64600 I
* 198.35.26.99/32         185.15.59.150                           14907 64600 I
* 198.35.26.112/28        185.15.59.150        0                  14907 I
* 198.35.26.112/32        185.15.59.150                           14907 64600 I
* 198.35.26.113/32        185.15.59.150                           14907 64600 I
* 198.35.26.192/32        185.15.59.150        1957               14907 I
* 198.35.26.193/32        185.15.59.150        1955               14907 I
* 198.35.26.194/32        185.15.59.150        21955              14907 I
* 198.35.26.196/31        185.15.59.150        1957               14907 I
* 198.35.26.198/31        185.15.59.150        21957              14907 I
* 198.35.26.200/31        185.15.59.150        21955              14907 I
* 198.35.26.202/31        185.15.59.150        1955               14907 I
* 198.35.26.204/31        185.15.59.150        2940               14907 I
* 198.35.26.208/31        185.15.59.150        2310               14907 I
* 198.35.26.224/29        185.15.59.150        1955               14907 I
* 198.35.26.240/28        185.15.59.150        1957               14907 I
* 198.35.27.0/24          185.15.59.150        0                  14907 I
* 198.73.209.0/24         185.15.59.150                           14907 11820 ?
* 206.53.202.0/24         185.15.59.150        1560               14907 I
* 206.126.236.0/22        185.15.59.150        1820               14907 I
* 206.197.187.0/24        185.15.59.150        1957               14907 I
* 206.223.116.0/23        185.15.59.150        1955               14907 I
* 206.223.118.0/23        185.15.59.150        1550               14907 I
* 208.80.152.0/23         185.15.59.150        0                  14907 I
* 208.80.152.224/28       185.15.59.150                           14907 64701 64600 I
* 208.80.152.240/28       185.15.59.150        1562               14907 I
* 208.80.153.0/27         185.15.59.150        1562               14907 I
* 208.80.153.32/27        185.15.59.150        1562               14907 I
* 208.80.153.64/27        185.15.59.150        1562               14907 I
* 208.80.153.96/27        185.15.59.150        1562               14907 I
* 208.80.153.176/31       185.15.59.150        1565               14907 I
* 208.80.153.178/31       185.15.59.150        1560               14907 I
* 208.80.153.184/29       185.15.59.150                           14907 64711 I
* 208.80.153.192/32       185.15.59.150        1565               14907 I
* 208.80.153.193/32       185.15.59.150        1560               14907 I
* 208.80.153.196/32       185.15.59.150        21560              14907 I
* 208.80.153.197/32       185.15.59.150                           14907 64701 I
* 208.80.153.198/32       185.15.59.150        1550               14907 I
* 208.80.153.200/31       185.15.59.150        1565               14907 I
* 208.80.153.202/31       185.15.59.150        1560               14907 I
* 208.80.153.204/31       185.15.59.150        2730               14907 I
* 208.80.153.206/31       185.15.59.150        21565              14907 I
* 208.80.153.208/31       185.15.59.150        21560              14907 I
* 208.80.153.210/31       185.15.59.150        1575               14907 I
* 208.80.153.212/31       185.15.59.150        1560               14907 I
* 208.80.153.214/31       185.15.59.150        2940               14907 I
* 208.80.153.218/31       185.15.59.150        1565               14907 I
* 208.80.153.220/31       185.15.59.150        1905               14907 I
* 208.80.153.222/31       185.15.59.150        1800               14907 I
* 208.80.153.224/27       185.15.59.150        0                  14907 I
* 208.80.153.224/28       185.15.59.150        0                  14907 I
* 208.80.153.224/32       185.15.59.150                           14907 64600 I
* 208.80.153.225/32       185.15.59.150                           14907 64600 I
* 208.80.153.226/32       185.15.59.150                           14907 64600 I
* 208.80.153.231/32       185.15.59.150        0                  14907 I
* 208.80.153.232/32       185.15.59.150                           14907 64600 I
* 208.80.153.240/28       185.15.59.150        0                  14907 I
* 208.80.153.240/32       185.15.59.150                           14907 64600 I
* 208.80.153.241/32       185.15.59.150                           14907 64600 I
* 208.80.153.252/32       185.15.59.150                           14907 64600 I
* 208.80.154.0/23         185.15.59.150        0                  14907 I
* 208.80.154.0/26         185.15.59.150        1822               14907 I
* 208.80.154.64/26        185.15.59.150        1822               14907 I
* 208.80.154.128/26       185.15.59.150        1822               14907 I
* 208.80.154.192/30       185.15.59.150        1823               14907 I
* 208.80.154.196/32       185.15.59.150        1820               14907 I
* 208.80.154.197/32       185.15.59.150        1823               14907 I
* 208.80.154.198/32       185.15.59.150        1800               14907 I
* 208.80.154.199/32       185.15.59.150        21820              14907 I
* 208.80.154.200/31       185.15.59.150        1820               14907 I
* 208.80.154.202/31       185.15.59.150        1823               14907 I
* 208.80.154.204/31       185.15.59.150        21820              14907 I
* 208.80.154.206/31       185.15.59.150        21823              14907 I
* 208.80.154.208/31       185.15.59.150        2040               14907 I
* 208.80.154.210/31       185.15.59.150        1820               14907 I
* 208.80.154.212/31       185.15.59.150        1823               14907 I
* 208.80.154.214/31       185.15.59.150        1900               14907 I
* 208.80.154.219/32       185.15.59.150                           14907 64700 I
* 208.80.154.220/31       185.15.59.150        1823               14907 I
* 208.80.154.224/27       185.15.59.150        0                  14907 I
* 208.80.154.224/28       185.15.59.150        0                  14907 I
* 208.80.154.224/32       185.15.59.150                           14907 64600 I
* 208.80.154.225/32       185.15.59.150                           14907 64600 I
* 208.80.154.226/32       185.15.59.150                           14907 64600 I
* 208.80.154.232/32       185.15.59.150                           14907 64600 I
* 208.80.154.238/32       185.15.59.150        0                  14907 I
* 208.80.154.240/28       185.15.59.150        0                  14907 I
* 208.80.154.240/32       185.15.59.150                           14907 64600 I
* 208.80.154.241/32       185.15.59.150                           14907 64600 I
* 208.80.154.242/32       185.15.59.150                           14907 64600 I
* 208.80.154.243/32       185.15.59.150                           14907 64600 I
* 208.80.154.244/32       185.15.59.150                           14907 64600 I
* 208.80.154.252/32       185.15.59.150                           14907 64600 I
* 208.80.155.0/27         185.15.59.150                           14907 64700 64600 I
* 208.80.155.64/28        185.15.59.150        1822               14907 I
* 208.80.155.96/27        185.15.59.150        1822               14907 I
* 208.115.136.0/23        185.15.59.150        1800               14907 I
* 216.66.30.88/30         185.15.59.150        1820               14907 I

# commit confirmed will be rolled back in 23 minutes
cmooney@asw1-by27-esams> show route receive-protocol bgp 2a02:ec80:300:fe08::1 table inet6.0 terse                            

Warning: License key missing; requires 'bgp' license


inet6.0: 314 destinations, 317 routes (313 active, 0 holddown, 2 hidden)
  Prefix		  Nexthop	       MED     Lclpref    AS path
* ::/0                    2a02:ec80:300:fe08::1                   14907 I
  2001:218:4000:5000::148/126
*                         2a02:ec80:300:fe08::1 3567              14907 I
  2001:418:0:5000::3b2/127
*                         2a02:ec80:300:fe08::1 1550              14907 I
  2001:418:0:5000::6fa/127
*                         2a02:ec80:300:fe08::1 1955              14907 I
  2001:418:16::110/127
*                         2a02:ec80:300:fe08::1 1820              14907 I
  2001:418:16::118/127
*                         2a02:ec80:300:fe08::1 1800              14907 I
* 2001:470:0:1c0::/64     2a02:ec80:300:fe08::1 1820              14907 I
* 2001:504:0:1::/64       2a02:ec80:300:fe08::1 1955              14907 I
* 2001:504:0:2::/64       2a02:ec80:300:fe08::1 1820              14907 I
* 2001:504:0:4::/64       2a02:ec80:300:fe08::1 1800              14907 I
* 2001:504:0:5::/64       2a02:ec80:300:fe08::1 1550              14907 I
* 2001:504:30::/64        2a02:ec80:300:fe08::1 1957              14907 I
* 2001:504:61::/64        2a02:ec80:300:fe08::1 1560              14907 I
* 2001:67c:608::/64       2a02:ec80:300:fe08::1 3                 14907 I
* 2001:67c:930::/48       2a02:ec80:300:fe08::1 0                 14907 I
  2001:688:0:4::2d0/127
*                         2a02:ec80:300:fe08::1 2702              14907 I
  2001:728:0:5000::164c/127
*                         2a02:ec80:300:fe08::1 2702              14907 I
* 2001:7f8:1::/64         2a02:ec80:300:fe08::1 3                 14907 I
* 2001:7f8:36::/64        2a02:ec80:300:fe08::1 2700              14907 I
* 2001:7f8:54:5::/64      2a02:ec80:300:fe08::1 2702              14907 I
* 2001:de8:4::/64         2a02:ec80:300:fe08::1 3565              14907 I
  2001:de8:12:100::/64
*                         2a02:ec80:300:fe08::1 3567              14907 I
* 2001:df2:e500::/48      2a02:ec80:300:fe08::1 0                 14907 I
  2001:df2:e500:1::/64
*                         2a02:ec80:300:fe08::1 3567              14907 I
  2001:df2:e500:101::/64
*                         2a02:ec80:300:fe08::1 3567              14907 I
  2001:df2:e500:201::/64
*                         2a02:ec80:300:fe08::1 3567              14907 I
  2001:df2:e500:ed1a::/64
*                         2a02:ec80:300:fe08::1 0                 14907 I
  2001:df2:e500:ed1a::/111
*                         2a02:ec80:300:fe08::1 0                 14907 I
  2001:df2:e500:ed1a::1/128             
*                         2a02:ec80:300:fe08::1                   14907 64600 I
  2001:df2:e500:ed1a::2/128
*                         2a02:ec80:300:fe08::1                   14907 64600 I
  2001:df2:e500:ed1a::3/128
*                         2a02:ec80:300:fe08::1                   14907 64600 I
  2001:df2:e500:ed1a::2:0/111
*                         2a02:ec80:300:fe08::1 0                 14907 I
  2001:df2:e500:ed1a::2:b/128
*                         2a02:ec80:300:fe08::1                   14907 64600 I
  2001:df2:e500:fe00::/64
*                         2a02:ec80:300:fe08::1 3567              14907 I
  2001:df2:e500:fe02::/64
*                         2a02:ec80:300:fe08::1 3565              14907 I
  2001:df2:e500:fe03::/64
*                         2a02:ec80:300:fe08::1 23565             14907 I
  2001:df2:e500:fe04::/64
*                         2a02:ec80:300:fe08::1 23567             14907 I
  2001:df2:e500:fe05::/64
*                         2a02:ec80:300:fe08::1 3567              14907 I
  2001:df2:e500:fe06::/64
*                         2a02:ec80:300:fe08::1 3565              14907 I
  2001:df2:e500:fe07::/64
*                         2a02:ec80:300:fe08::1 4550              14907 I
  2001:df2:e500:ffff::1/128
*                         2a02:ec80:300:fe08::1 23565             14907 I
  2001:df2:e500:ffff::3/128
*                         2a02:ec80:300:fe08::1 3567              14907 I
  2001:df2:e500:ffff::4/128
*                         2a02:ec80:300:fe08::1 3565              14907 I
  2001:df5:b800:bb00::/64
*                         2a02:ec80:300:fe08::1 3567              14907 I
  2001:1900:2100::998/126
*                         2a02:ec80:300:fe08::1 1800              14907 I
  2001:1900:2100::a98/126
*                         2a02:ec80:300:fe08::1 1957              14907 I
  2001:1900:2100::a9c/126
*                         2a02:ec80:300:fe08::1 1820              14907 I
  2001:1900:2100::4b40/126
*                         2a02:ec80:300:fe08::1 1560              14907 I
  2001:2000:3080:a98::/64
*                         2a02:ec80:300:fe08::1 1823              14907 I
  2001:2000:3080:a9a::/64
*                         2a02:ec80:300:fe08::1 1957              14907 I
  2001:2000:3080:af4::/64
*                         2a02:ec80:300:fe08::1 1565              14907 I
  2001:2000:3080:15b5::/126             
*                         2a02:ec80:300:fe08::1 3565              14907 I
  2001:2035:0:699::/126
*                         2a02:ec80:300:fe08::1 3                 14907 I
  2001:2035:0:cf1::/126
*                         2a02:ec80:300:fe08::1 2700              14907 I
  2405:2000:800::18/126
*                         2a02:ec80:300:fe08::1 3567              14907 I
* 2620:0:860::/48         2a02:ec80:300:fe08::1 0                 14907 I
* 2620:0:860:1::/64       2a02:ec80:300:fe08::1 1562              14907 I
* 2620:0:860:2::/64       2a02:ec80:300:fe08::1 1562              14907 I
* 2620:0:860:3::/64       2a02:ec80:300:fe08::1 1562              14907 I
* 2620:0:860:4::/64       2a02:ec80:300:fe08::1 1562              14907 I
* 2620:0:860:101::/64     2a02:ec80:300:fe08::1 1562              14907 I
* 2620:0:860:102::/64     2a02:ec80:300:fe08::1 1562              14907 I
* 2620:0:860:103::/64     2a02:ec80:300:fe08::1 1562              14907 I
* 2620:0:860:104::/64     2a02:ec80:300:fe08::1 1562              14907 I
* 2620:0:860:118::/64     2a02:ec80:300:fe08::1                   14907 64711 I
* 2620:0:860:122::/64     2a02:ec80:300:fe08::1 1562              14907 I
* 2620:0:860:130::/64     2a02:ec80:300:fe08::1 1565              14907 I
* 2620:0:860:131::/64     2a02:ec80:300:fe08::1 1560              14907 I
* 2620:0:860:13f::/64     2a02:ec80:300:fe08::1                   14907 64711 I
* 2620:0:860:201::/64     2a02:ec80:300:fe08::1 1562              14907 I
  2620:0:860:300:2f5:5178:8fd8:ad00/122
*                         2a02:ec80:300:fe08::1                   14907 64607 I
  2620:0:860:300:4070:19a0:cb8d:fc0/122
*                         2a02:ec80:300:fe08::1                   14907 64607 I
  2620:0:860:300:42ee:ed84:91a1:7e40/122
*                         2a02:ec80:300:fe08::1                   14907 64607 I
  2620:0:860:300:6bfb:4166:893e:2a00/122
*                         2a02:ec80:300:fe08::1                   14907 64607 I
  2620:0:860:300:7eda:b3e9:c49f:4200/122
*                         2a02:ec80:300:fe08::1                   14907 64607 I
  2620:0:860:300:98fd:ab0f:a5b6:2b00/122
*                         2a02:ec80:300:fe08::1                   14907 64607 I
  2620:0:860:300:bfc9:7fbd:3a67:7ec0/122
*                         2a02:ec80:300:fe08::1                   14907 64607 I
  2620:0:860:300:ec47:8d7f:a36a:4080/122
*                         2a02:ec80:300:fe08::1                   14907 64607 I
  2620:0:860:302:cd1a:dd9d:d3d3:2780/122
*                         2a02:ec80:300:fe08::1                   14907 64608 I
  2620:0:860:302:dce5:64c3:ebd6:33c0/122
*                         2a02:ec80:300:fe08::1                   14907 64608 I
  2620:0:860:babe:316c:46ce:a28b:1280/122
*                         2a02:ec80:300:fe08::1                   14907 64604 I
  2620:0:860:babe:6daa:bbed:4001:c040/122
*                         2a02:ec80:300:fe08::1                   14907 64604 I
  2620:0:860:babe:9763:6d0c:9ec5:80/122
*                         2a02:ec80:300:fe08::1                   14907 64604 I
  2620:0:860:babe:dba4:4732:26b4:5500/122
*                         2a02:ec80:300:fe08::1                   14907 64604 I
  2620:0:860:cabe:5e6:a82f:f8c3:b240/122
*                         2a02:ec80:300:fe08::1                   14907 64602 I
  2620:0:860:cabe:ee3:4b71:55a9:1980/122
*                         2a02:ec80:300:fe08::1                   14907 64602 I
  2620:0:860:cabe:f82:925a:a0ca:38c0/122
*                         2a02:ec80:300:fe08::1                   14907 64602 I
  2620:0:860:cabe:124a:a30b:7d97:600/122
*                         2a02:ec80:300:fe08::1                   14907 64602 I
  2620:0:860:cabe:1716:f6c:10e4:4e00/122
*                         2a02:ec80:300:fe08::1                   14907 64602 I
  2620:0:860:cabe:1a71:4695:f1ec:f700/122
*                         2a02:ec80:300:fe08::1                   14907 64602 I
  2620:0:860:cabe:2263:a6bf:79d8:abc0/122
*                         2a02:ec80:300:fe08::1                   14907 64602 I
  2620:0:860:cabe:28e8:8c5a:3944:5c80/122
*                         2a02:ec80:300:fe08::1                   14907 64602 I
  2620:0:860:cabe:2ff5:cf90:ec9e:4b40/122
*                         2a02:ec80:300:fe08::1                   14907 64602 I
  2620:0:860:cabe:37ca:8dec:1af6:2a80/122
*                         2a02:ec80:300:fe08::1                   14907 64602 I
  2620:0:860:cabe:6722:1a79:ac53:7680/122
*                         2a02:ec80:300:fe08::1                   14907 64602 I
  2620:0:860:cabe:6a1a:9cfd:4e0d:b40/122
*                         2a02:ec80:300:fe08::1                   14907 64602 I
  2620:0:860:cabe:6b98:1a02:5e8:f200/122
*                         2a02:ec80:300:fe08::1                   14907 64602 I
  2620:0:860:cabe:7491:5c3b:5e90:68c0/122
*                         2a02:ec80:300:fe08::1                   14907 64602 I
  2620:0:860:cabe:7623:defb:5ecf:cdc0/122
*                         2a02:ec80:300:fe08::1                   14907 64602 I
  2620:0:860:cabe:877b:fad0:ab10:d0c0/122
*                         2a02:ec80:300:fe08::1                   14907 64602 I
  2620:0:860:cabe:a0a6:a378:8311:2d40/122
*                         2a02:ec80:300:fe08::1                   14907 64602 I
  2620:0:860:cabe:a8f6:3e6b:2ac8:f000/122
*                         2a02:ec80:300:fe08::1                   14907 64602 I
  2620:0:860:cabe:af9d:86e8:31bd:880/122
*                         2a02:ec80:300:fe08::1                   14907 64602 I
  2620:0:860:cabe:ba75:89d6:7d8a:3780/122
*                         2a02:ec80:300:fe08::1                   14907 64602 I
  2620:0:860:cabe:d47c:5fb1:731f:8400/122
*                         2a02:ec80:300:fe08::1                   14907 64602 I
  2620:0:860:cabe:db1a:456f:d70:c40/122
*                         2a02:ec80:300:fe08::1                   14907 64602 I
  2620:0:860:ed1a::/64
*                         2a02:ec80:300:fe08::1 0                 14907 I
  2620:0:860:ed1a::/111
*                         2a02:ec80:300:fe08::1 0                 14907 I
  2620:0:860:ed1a::1/128
*                         2a02:ec80:300:fe08::1                   14907 64600 I
  2620:0:860:ed1a::2/128
*                         2a02:ec80:300:fe08::1                   14907 64600 I
  2620:0:860:ed1a::9/128
*                         2a02:ec80:300:fe08::1                   14907 64600 I
  2620:0:860:ed1a::2:0/111
*                         2a02:ec80:300:fe08::1 0                 14907 I
  2620:0:860:ed1a::2:b/128
*                         2a02:ec80:300:fe08::1                   14907 64600 I
  2620:0:860:fe00::/64
*                         2a02:ec80:300:fe08::1 1565              14907 I
  2620:0:860:fe01::/64
*                         2a02:ec80:300:fe08::1 1905              14907 I
  2620:0:860:fe02::/64
*                         2a02:ec80:300:fe08::1 1800              14907 I
  2620:0:860:fe03::/64
*                         2a02:ec80:300:fe08::1 1575              14907 I
  2620:0:860:fe04::/64
*                         2a02:ec80:300:fe08::1 1560              14907 I
  2620:0:860:fe05::/64
*                         2a02:ec80:300:fe08::1 21565             14907 I
  2620:0:860:fe06::/64
*                         2a02:ec80:300:fe08::1 21560             14907 I
  2620:0:860:fe08::/64
*                         2a02:ec80:300:fe08::1 2940              14907 I
  2620:0:860:fe0a::/64
*                         2a02:ec80:300:fe08::1 2730              14907 I
  2620:0:860:ffff::1/128
*                         2a02:ec80:300:fe08::1 1565              14907 I
  2620:0:860:ffff::2/128
*                         2a02:ec80:300:fe08::1 1560              14907 I
  2620:0:860:ffff::5/128
*                         2a02:ec80:300:fe08::1 1550              14907 I
  2620:0:860:ffff::6/128
*                         2a02:ec80:300:fe08::1 21560             14907 I
* 2620:0:861::/48         2a02:ec80:300:fe08::1 0                 14907 I
* 2620:0:861:1::/64       2a02:ec80:300:fe08::1 1822              14907 I
* 2620:0:861:2::/64       2a02:ec80:300:fe08::1 1822              14907 I
* 2620:0:861:3::/64       2a02:ec80:300:fe08::1 1822              14907 I
* 2620:0:861:4::/64       2a02:ec80:300:fe08::1 1822              14907 I
* 2620:0:861:100::/64     2a02:ec80:300:fe08::1                   14907 64810 I
* 2620:0:861:101::/64     2a02:ec80:300:fe08::1 1822              14907 I
* 2620:0:861:102::/64     2a02:ec80:300:fe08::1 1822              14907 I
* 2620:0:861:103::/64     2a02:ec80:300:fe08::1 1822              14907 I
* 2620:0:861:104::/64     2a02:ec80:300:fe08::1 1822              14907 I
* 2620:0:861:105::/64     2a02:ec80:300:fe08::1 1822              14907 I
* 2620:0:861:106::/64     2a02:ec80:300:fe08::1 1822              14907 I
* 2620:0:861:107::/64     2a02:ec80:300:fe08::1 1822              14907 I
* 2620:0:861:108::/64     2a02:ec80:300:fe08::1 1822              14907 I
* 2620:0:861:109::/64     2a02:ec80:300:fe08::1                   14907 64810 I
* 2620:0:861:10a::/64     2a02:ec80:300:fe08::1                   14907 64810 I
* 2620:0:861:10b::/64     2a02:ec80:300:fe08::1                   14907 64810 I
* 2620:0:861:10d::/64     2a02:ec80:300:fe08::1                   14907 64810 I
* 2620:0:861:10e::/64     2a02:ec80:300:fe08::1                   14907 64810 I
* 2620:0:861:10f::/64     2a02:ec80:300:fe08::1                   14907 64810 I
* 2620:0:861:111::/64     2a02:ec80:300:fe08::1                   14907 64810 I
* 2620:0:861:112::/64     2a02:ec80:300:fe08::1                   14907 64810 I
* 2620:0:861:114::/64     2a02:ec80:300:fe08::1                   14907 64810 I
* 2620:0:861:115::/64     2a02:ec80:300:fe08::1                   14907 64810 I
* 2620:0:861:116::/64     2a02:ec80:300:fe08::1                   14907 64810 I
* 2620:0:861:117::/64     2a02:ec80:300:fe08::1 1822              14907 I
* 2620:0:861:118::/64     2a02:ec80:300:fe08::1                   14907 64710 I
* 2620:0:861:119::/64     2a02:ec80:300:fe08::1 1822              14907 I
  2620:0:861:11b::1/128
*                         2a02:ec80:300:fe08::1                   14907 64810 I
  2620:0:861:11b::2/128
*                         2a02:ec80:300:fe08::1                   14907 64810 I
  2620:0:861:11b::3/128
*                         2a02:ec80:300:fe08::1                   14907 64810 I
  2620:0:861:11b::4/128
*                         2a02:ec80:300:fe08::1                   14907 64810 I
  2620:0:861:11b::5/128
*                         2a02:ec80:300:fe08::1                   14907 64810 I
  2620:0:861:11b::7/128
*                         2a02:ec80:300:fe08::1                   14907 64810 I
  2620:0:861:11b::8/128
*                         2a02:ec80:300:fe08::1                   14907 64810 I
  2620:0:861:11b::9/128
*                         2a02:ec80:300:fe08::1                   14907 64810 I
  2620:0:861:11b::251/128
*                         2a02:ec80:300:fe08::1                   14907 64710 I
  2620:0:861:11b::252/128
*                         2a02:ec80:300:fe08::1                   14907 64710 I
  2620:0:861:11b::253/128
*                         2a02:ec80:300:fe08::1                   14907 64710 I
  2620:0:861:11b::254/128
*                         2a02:ec80:300:fe08::1                   14907 64710 4264710003 I
  2620:0:861:11b::255/128
*                         2a02:ec80:300:fe08::1                   14907 64710 4264710004 I
* 2620:0:861:11c::/64     2a02:ec80:300:fe08::1                   14907 64710 4264710003 I
* 2620:0:861:11d::/64     2a02:ec80:300:fe08::1                   14907 64710 4264710004 I
* 2620:0:861:11e::/64     2a02:ec80:300:fe08::1                   14907 64710 I
* 2620:0:861:11f::/64     2a02:ec80:300:fe08::1                   14907 64710 I
* 2620:0:861:202::/64     2a02:ec80:300:fe08::1 1822              14907 I
  2620:0:861:300:233d:9516:7794:b080/122
*                         2a02:ec80:300:fe08::1                   14907 64810 64606 I
  2620:0:861:300:27e1:d164:3470:4d40/122
*                         2a02:ec80:300:fe08::1                   14907 64606 I
  2620:0:861:300:2e5e:147b:e43b:e580/122
*                         2a02:ec80:300:fe08::1                   14907 64810 64606 I
  2620:0:861:300:6f3b:651e:b3d8:3d40/122
*                         2a02:ec80:300:fe08::1                   14907 64606 I
  2620:0:861:300:9c11:509e:dd28:5780/122
*                         2a02:ec80:300:fe08::1                   14907 64810 64606 I
  2620:0:861:300:a223:b79f:83e0:a640/122
*                         2a02:ec80:300:fe08::1                   14907 64606 I
  2620:0:861:300:ae95:5828:1aaf:42c0/122
*                         2a02:ec80:300:fe08::1                   14907 64810 64606 I
  2620:0:861:300:df9c:3303:568c:4c80/122
*                         2a02:ec80:300:fe08::1                   14907 64606 I
  2620:0:861:302:1508:3a63:3151:9840/122
*                         2a02:ec80:300:fe08::1                   14907 64609 I
  2620:0:861:302:22de:3a32:fa8a:bcc0/122
*                         2a02:ec80:300:fe08::1                   14907 64609 I
  2620:0:861:302:278e:a9a9:d3bd:c040/122
*                         2a02:ec80:300:fe08::1                   14907 64609 I
  2620:0:861:302:32a6:61fe:4eeb:d0c0/122
*                         2a02:ec80:300:fe08::1                   14907 64810 64609 I
  2620:0:861:302:b0bc:e504:eb5e:e980/122
*                         2a02:ec80:300:fe08::1                   14907 64609 I
  2620:0:861:302:b568:55e1:16d7:d280/122
*                         2a02:ec80:300:fe08::1                   14907 64810 64609 I
  2620:0:861:302:d175:515d:1baf:e40/122
*                         2a02:ec80:300:fe08::1                   14907 64810 64609 I
  2620:0:861:302:f380:aae4:1a76:e6c0/122
*                         2a02:ec80:300:fe08::1                   14907 64810 64609 I
  2620:0:861:305:2d23:2318:32d0:5d00/122
*                         2a02:ec80:300:fe08::1                   14907 64610 I
  2620:0:861:305:9f45:2ce3:4648:3c80/122
*                         2a02:ec80:300:fe08::1                   14907 64610 I
  2620:0:861:babe:710a:708f:1ba4:fac0/122
*                         2a02:ec80:300:fe08::1                   14907 64603 I
  2620:0:861:babe:92f3:bb8e:59e0:9700/122
*                         2a02:ec80:300:fe08::1                   14907 64603 I
  2620:0:861:babe:ae01:ebe3:2ae2:0/122
*                         2a02:ec80:300:fe08::1                   14907 64603 I
  2620:0:861:babe:b457:a930:98ea:7640/122
*                         2a02:ec80:300:fe08::1                   14907 64603 I
  2620:0:861:cabe:6c4:334c:e7b7:17c0/122
*                         2a02:ec80:300:fe08::1                   14907 64601 I
  2620:0:861:cabe:79b:f9f9:f5d5:c700/122
*                         2a02:ec80:300:fe08::1                   14907 64601 I
  2620:0:861:cabe:857:5359:80ea:6b00/122
*                         2a02:ec80:300:fe08::1                   14907 64601 I
  2620:0:861:cabe:2828:2a28:7cd0:de00/122
*                         2a02:ec80:300:fe08::1                   14907 64601 I
  2620:0:861:cabe:2961:2acc:b9dc:6600/122
*                         2a02:ec80:300:fe08::1                   14907 64601 I
  2620:0:861:cabe:3d25:6fb9:6ed7:1d80/122
*                         2a02:ec80:300:fe08::1                   14907 64601 I
  2620:0:861:cabe:43bd:eeed:f4d4:d840/122
*                         2a02:ec80:300:fe08::1                   14907 64601 I
  2620:0:861:cabe:4b88:bfe2:df44:2800/122
*                         2a02:ec80:300:fe08::1                   14907 64601 I
  2620:0:861:cabe:526c:e3ed:4e61:b240/122
*                         2a02:ec80:300:fe08::1                   14907 64601 I
  2620:0:861:cabe:544d:441b:2e88:a640/122
*                         2a02:ec80:300:fe08::1                   14907 64810 64601 I
  2620:0:861:cabe:5c4a:1f61:84c5:a100/122
*                         2a02:ec80:300:fe08::1                   14907 64601 I
  2620:0:861:cabe:847d:b4ea:704e:f980/122
*                         2a02:ec80:300:fe08::1                   14907 64601 I
  2620:0:861:cabe:8ee4:d43b:7a92:c80/122
*                         2a02:ec80:300:fe08::1                   14907 64601 I
  2620:0:861:cabe:8fd5:c154:2a91:fac0/122
*                         2a02:ec80:300:fe08::1                   14907 64601 I
  2620:0:861:cabe:a857:76e9:4001:5f80/122
*                         2a02:ec80:300:fe08::1                   14907 64601 I
  2620:0:861:cabe:b4ba:5e94:c142:6f80/122
*                         2a02:ec80:300:fe08::1                   14907 64601 I
  2620:0:861:cabe:c0c3:c710:f7:8280/122
*                         2a02:ec80:300:fe08::1                   14907 64601 I
  2620:0:861:cabe:d120:eaff:1f0d:6800/122
*                         2a02:ec80:300:fe08::1                   14907 64810 64601 I
  2620:0:861:cabe:e7a3:ffcd:aa79:f580/122
*                         2a02:ec80:300:fe08::1                   14907 64601 I
  2620:0:861:cabe:ed16:f59c:6f0b:1e80/122
*                         2a02:ec80:300:fe08::1                   14907 64601 I
  2620:0:861:cabe:eeaa:7719:464:53c0/122
*                         2a02:ec80:300:fe08::1                   14907 64601 I
  2620:0:861:cabe:ef08:ec86:df2b:bfc0/122
*                         2a02:ec80:300:fe08::1                   14907 64601 I
  2620:0:861:cabe:f071:e804:3296:5500/122
*                         2a02:ec80:300:fe08::1                   14907 64601 I
  2620:0:861:cabe:f244:a3bd:c8d:d040/122
*                         2a02:ec80:300:fe08::1                   14907 64601 I
  2620:0:861:ed1a::/64
*                         2a02:ec80:300:fe08::1 0                 14907 I
  2620:0:861:ed1a::/111
*                         2a02:ec80:300:fe08::1 0                 14907 I
  2620:0:861:ed1a::1/128
*                         2a02:ec80:300:fe08::1                   14907 64600 I
  2620:0:861:ed1a::2/128
*                         2a02:ec80:300:fe08::1                   14907 64600 I
  2620:0:861:ed1a::9/128
*                         2a02:ec80:300:fe08::1                   14907 64600 I
  2620:0:861:ed1a::2:0/111
*                         2a02:ec80:300:fe08::1 0                 14907 I
  2620:0:861:ed1a::2:b/128
*                         2a02:ec80:300:fe08::1                   14907 64600 I
  2620:0:861:ed1a::3:241/128
*                         2a02:ec80:300:fe08::1                   14907 64600 I
  2620:0:861:fe00::/64
*                         2a02:ec80:300:fe08::1 1823              14907 I
  2620:0:861:fe02::/64
*                         2a02:ec80:300:fe08::1 2040              14907 I
  2620:0:861:fe03::/64
*                         2a02:ec80:300:fe08::1 1823              14907 I
  2620:0:861:fe04::/64
*                         2a02:ec80:300:fe08::1 21820             14907 I
  2620:0:861:fe05::/64
*                         2a02:ec80:300:fe08::1 21823             14907 I
  2620:0:861:fe06::/64
*                         2a02:ec80:300:fe08::1 1900              14907 I
  2620:0:861:fe07::/64
*                         2a02:ec80:300:fe08::1 1820              14907 I
  2620:0:861:fe08::/64
*                         2a02:ec80:300:fe08::1 1823              14907 I
  2620:0:861:fe09::/64
*                         2a02:ec80:300:fe08::1 1820              14907 I
  2620:0:861:fe0a::/64
*                         2a02:ec80:300:fe08::1 1823              14907 I
  2620:0:861:ffff::1/128                
*                         2a02:ec80:300:fe08::1 1820              14907 I
  2620:0:861:ffff::2/128
*                         2a02:ec80:300:fe08::1 1823              14907 I
  2620:0:861:ffff::5/128
*                         2a02:ec80:300:fe08::1 1800              14907 I
  2620:0:861:ffff::6/128
*                         2a02:ec80:300:fe08::1 21820             14907 I
  2620:0:862:fe07::/64
*                         2a02:ec80:300:fe08::1 5003              14907 I
* 2620:0:863::/48         2a02:ec80:300:fe08::1 0                 14907 I
* 2620:0:863:1::/64       2a02:ec80:300:fe08::1 1957              14907 I
* 2620:0:863:101::/64     2a02:ec80:300:fe08::1 1957              14907 I
* 2620:0:863:201::/64     2a02:ec80:300:fe08::1 1957              14907 I
  2620:0:863:ed1a::/64
*                         2a02:ec80:300:fe08::1 0                 14907 I
  2620:0:863:ed1a::/111
*                         2a02:ec80:300:fe08::1 0                 14907 I
  2620:0:863:ed1a::1/128
*                         2a02:ec80:300:fe08::1                   14907 64600 I
  2620:0:863:ed1a::2/128
*                         2a02:ec80:300:fe08::1                   14907 64600 I
  2620:0:863:ed1a::3/128
*                         2a02:ec80:300:fe08::1                   14907 64600 I
  2620:0:863:ed1a::2:0/111
*                         2a02:ec80:300:fe08::1 0                 14907 I
  2620:0:863:ed1a::2:b/128
*                         2a02:ec80:300:fe08::1                   14907 64600 I
  2620:0:863:fe00::/64
*                         2a02:ec80:300:fe08::1 1957              14907 I
  2620:0:863:fe02::/64
*                         2a02:ec80:300:fe08::1 2310              14907 I
  2620:0:863:fe04::/64
*                         2a02:ec80:300:fe08::1 2940              14907 I
  2620:0:863:fe05::/64
*                         2a02:ec80:300:fe08::1 21957             14907 I
  2620:0:863:fe06::/64
*                         2a02:ec80:300:fe08::1 21955             14907 I
  2620:0:863:fe07::/64
*                         2a02:ec80:300:fe08::1 1955              14907 I
  2620:0:863:ffff::1/128
*                         2a02:ec80:300:fe08::1 1957              14907 I
  2620:0:863:ffff::2/128
*                         2a02:ec80:300:fe08::1 1955              14907 I
  2620:0:863:ffff::6/128
*                         2a02:ec80:300:fe08::1 21955             14907 I
  2620:0:1cff:dead:beee::11c8/127       
*                         2a02:ec80:300:fe08::1 1550              14907 I
  2620:0:1cff:dead:beee::12b8/127
*                         2a02:ec80:300:fe08::1 1820              14907 I
  2620:62:c000::200:149:0/125
*                         2a02:ec80:300:fe08::1 1955              14907 I
  2a01:3e0:ff50::3c/126
*                         2a02:ec80:300:fe08::1 2700              14907 I
  2a02:ec80:300:fe04::/64
*                         2a02:ec80:300:fe08::1 3                 14907 I
  2a02:ec80:300:fe05::/64
                          2a02:ec80:300:fe08::1 3                 14907 I
  2a02:ec80:300:fe09::/64
*                         2a02:ec80:300:fe08::1 5003              14907 I
  2a02:ec80:300:fe0a::/64
*                         2a02:ec80:300:fe08::1 5003              14907 I
  2a02:ec80:300:ffff::1/128
*                         2a02:ec80:300:fe08::1 3                 14907 I
* 2a02:ec80:600::/48      2a02:ec80:300:fe08::1 0                 14907 I
  2a02:ec80:600:1::/64
*                         2a02:ec80:300:fe08::1                   14907 4265006001 I
  2a02:ec80:600:2::/64
*                         2a02:ec80:300:fe08::1                   14907 4265006002 I
  2a02:ec80:600:101::/64
*                         2a02:ec80:300:fe08::1                   14907 4265006001 I
  2a02:ec80:600:102::/64
*                         2a02:ec80:300:fe08::1                   14907 4265006002 I
  2a02:ec80:600:ed1a::/64
*                         2a02:ec80:300:fe08::1 0                 14907 I
  2a02:ec80:600:ed1a::1/128
*                         2a02:ec80:300:fe08::1                   14907 4265006001 64600 I
  2a02:ec80:600:ed1a::2/128
*                         2a02:ec80:300:fe08::1                   14907 4265006001 64600 I
  2a02:ec80:600:ed1a::3/128
*                         2a02:ec80:300:fe08::1                   14907 4265006001 64600 I
  2a02:ec80:600:ed1a::2:b/128
*                         2a02:ec80:300:fe08::1                   14907 4265006002 64600 I
  2a02:ec80:600:fe01::/64
*                         2a02:ec80:300:fe08::1 2700              14907 I
  2a02:ec80:600:fe02::/64
*                         2a02:ec80:300:fe08::1                   14907 4265006001 I
  2a02:ec80:600:fe03::/64
*                         2a02:ec80:300:fe08::1                   14907 4265006002 I
  2a02:ec80:600:fe04::/64
*                         2a02:ec80:300:fe08::1 2788              14907 I
  2a02:ec80:600:fe05::/64
*                         2a02:ec80:300:fe08::1 2702              14907 I
  2a02:ec80:600:fe06::/64
*                         2a02:ec80:300:fe08::1 2700              14907 I
  2a02:ec80:600:fe07::/64
*                         2a02:ec80:300:fe08::1 2700              14907 I
  2a02:ec80:600:fe08::/64
*                         2a02:ec80:300:fe08::1 2702              14907 I
  2a02:ec80:600:fe09::/64
*                         2a02:ec80:300:fe08::1 2702              14907 I
  2a02:ec80:600:ffff::1/128
*                         2a02:ec80:300:fe08::1 2700              14907 I
  2a02:ec80:600:ffff::2/128
*                         2a02:ec80:300:fe08::1 2702              14907 I
  2a02:ec80:600:ffff::3/128
*                         2a02:ec80:300:fe08::1                   14907 4265006001 65000 I
  2a02:ec80:600:ffff::4/128
*                         2a02:ec80:300:fe08::1                   14907 4265006001 I
  2a02:ec80:600:ffff::5/128
*                         2a02:ec80:300:fe08::1                   14907 4265006002 I

Event Timeline

cmooney triaged this task as Low priority.
cmooney updated the task description. (Show Details)

In terms of Pybal or Anycast routes etc. we can possibly not send those. If the overall approach seems good we can weigh up exactly what is useful to send (certainly subnets associated with private vlans and things like loopback IPs).

This should also result in more optimal routing as switches will send traffic to the CR with the best path to the remote internal destinations. i.e. if CR1 has the best path to eqiad then the switches will see that, and traffic won't get sent to CR2 before taking that path anyway.

On the resiliency side, this protects us from a double failure: the cr1-cr2 link to fail as well as a transport link. Low risk but still a risk.
I agree on the optimal routing, it removes one unnecessary hop in ~50% of the cases.

Some downsides I can think off: additional config, more complex to troubleshot (more prefixes in the routing table), more dynamic (of course by design, but risks of saturating the RIB with a miss-config)
Routing convergence is probably not a factor here.

It would make sens to deploy it, but while trying to keep it more lean:

we can instead use a tailored one if we decide to deploy

So far we only filter on the exporter side everywhere, I think it's better to not start filtering on both sides

set policy-options policy-statement Switch_out term from_bgp from as-path core_and_local_LVS
set policy-options policy-statement Switch_out term from_bgp from as-path anycast

Those 2 lines are not needed as from as-path from_remote_confed already takes care of helping pick the best transport link.

I'm not sure the prefix-limit is needed.
Leaving it as it means only log, which at best could trigger an LibreNMS "log based" alert, but the time someone acts on it it might be too late (we already do for peering).
Adding the teardown config knob to actually take the BGP session down, would only be better than saturating the RIB with static routes in both directions (otherwise BGP is down and hosts are unreachable).

We need to also harmonize it everywhere, including core routers to spines BGP sessions, where we currently only export default.

Some downsides I can think off: additional config, more complex to troubleshot (more prefixes in the routing table), more dynamic (of course by design, but risks of saturating the RIB with a miss-config)
Routing convergence is probably not a factor here.

Yeah I think the overall numbers are low enough that convergence isn't an issue. The other risk factors certainly all should be considered but I think none are a big enough worry that would put me off.

So far we only filter on the exporter side everywhere, I think it's better to not start filtering on both sides

In drmrs we were filtering inbound, which I'd copied to the new asw's in esams. But I agree filtering outbound on our internal links is enough we can remove that.

I'm not sure the prefix-limit is needed.

I think it's a sensible precaution on any device that can't fit a full table when peering to one which has. Don't feel that strongly but I don't see that it hurts too much. I'd left out the 'teardown' in my example that's what I had in mind.

Adding the teardown config knob to actually take the BGP session down, would only be better than saturating the RIB with static routes in both directions (otherwise BGP is down and hosts are unreachable).

What I've seen in the past when observing (or making) such mistakes is the device doesn't recover cleanly at all from a total TCAM exhaustion. TBH I've no info on if the Juniper QFX's (or Broadcom ASIC) have this problem or not, and it's been a while. But when I've encountered this before the fall-out was total control-plane meltdown requiring reboot, far worse than if BGP shut itself down when it triggered.

We need to also harmonize it everywhere, including core routers to spines BGP sessions, where we currently only export default.

Indeed yep on all CR -> Switch links we should do it. Probably also to the cloudsw (for consistency). One thing I'm unsure about is on BGP from ASW to MR like we have in esams. I think possibly safer not to touch that for now, and given the topology I think the risk is lower.

Cool, thanks for the details, makes sens to use prefix-limit with teardown then, maybe some timeout so it automatically recovers and double check our alerting so we're aware when a sessions has been brought down.

set policy-options policy-statement Switch_out term from_bgp from as-path core_and_local_LVS
set policy-options policy-statement Switch_out term from_bgp from as-path anycast

Those 2 lines are not needed as from as-path from_remote_confed already takes care of helping pick the best transport link.

We need these at least to announce a primary LVS routes known locally, to prevent a switch picking a backup path instead.

Announcing all the remote ranges but not the local ones seems slightly counter-intuitive, although you are correct it's not essential. On a wider policy level I think it's simpler to just to say "announce all internal routes", open to discuss but it seems like an easier rule of thumb to work with.

My understanding is that the LVS term is a fix for that one issue (now that I'm aware of T348446).
If we start adding anycast then we need to add all the possible server advertised ASNs to not risk forgetting any of them in the longer run and standardize the term across all sites. So either none (no anycast) or all (either a list of ASN or a regex range of all the private ASNs).
Does that make sens?

Mentioned in SAL (#wikimedia-operations) [2023-10-27T14:19:53Z] <topranks> announcing internal core routes to esams asw's to test policy T344547

Change 969367 had a related patch set uploaded (by Cathal Mooney; author: Cathal Mooney):

[operations/homer/public@master] Change core router config to export internal routes to Switches

https://gerrit.wikimedia.org/r/969367

Above patch reflects my thinking on the best approach for this. I've taken the approach that we should announce all our internal routes to the switches.

There are probably some cases where having a particular local route (like ranges announced by local K8s hosts) isn't of particular benefit, the switch can as well follow the default (doesn't matter which CR it hits). But the complexity of

The other thing I've included is to set the BGP MED outbound to the OSPF cost. This ensures switches will send traffic to the preferred CR for internal destinations, preventing packets being sent to, for instance, CR1 and then being sent to CR2.

FWIW in my original config for this I had terms to match routes redistributed into BGP locally and announced in IBGP, or between confed sub-as's. We have no such routes at the moment, however. All our routes either originate in EBGP, or get redistributed from direct to OSPF (not direct to BGP). So I've removed the terms related to those If we change things in future we can re-add terms similar to these:

set policy-options policy-statement Switch_out term remote_bgp from protocol bgp
set policy-options policy-statement Switch_out term remote_bgp from as-path from_remote_confed
set policy-options policy-statement Switch_out term remote_bgp then metric igp
set policy-options policy-statement Switch_out term remote_bgp then accept
set policy-options policy-statement Switch_out term local_ibgp from protocol bgp
set policy-options policy-statement Switch_out term local_ibgp from as-path-calc-length 0 equal
set policy-options policy-statement Switch_out term local_ibgp then metric igp
set policy-options policy-statement Switch_out term local_ibgp then accept

There is also the matter of CR directly connected routes. All such interfaces are in OSPF currently, and thus exported by that term. If we ever stop distributing them in OSPF we may need to add a term for "protocol direct."

Change 969367 merged by jenkins-bot:

[operations/homer/public@master] Change core router config to export internal routes to Switches

https://gerrit.wikimedia.org/r/969367

Mentioned in SAL (#wikimedia-operations) [2023-11-01T19:44:52Z] <topranks> adjusting routes announced to L3 switches in codfw T344547

Mentioned in SAL (#wikimedia-operations) [2023-11-01T20:23:16Z] <topranks> adjusting routes announced to L3 switches in esams T344547

Mentioned in SAL (#wikimedia-operations) [2023-11-01T20:49:53Z] <topranks> configure esams switches to load-share default across CRs T344547

Mentioned in SAL (#wikimedia-operations) [2023-11-01T21:14:43Z] <topranks> adjust BGP policy out to L3 switches on remaining CRs T344547

cmooney closed this task as Resolved.EditedNov 1 2023, 10:03 PM
cmooney claimed this task.

Patch is merged everywhere. Looks ok.

For instance switch in esams connected to backup LVS now sends traffic to primary LVS via CRs:

cmooney@asw1-bw27-esams> show route 185.15.59.240/32 exact    

inet.0: 642 destinations, 1259 routes (642 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

185.15.59.240/32   *[BGP/170] 00:00:07, MED 0, localpref 100, from 185.15.59.156
                      AS path: 14907 4265003002 64600 I, validation-state: unverified
                       to 185.15.59.156 via et-0/0/48.0
                    >  to 185.15.59.158 via et-0/0/50.0
                    [BGP/170] 00:43:12, MED 0, localpref 100
                      AS path: 14907 4265003002 64600 I, validation-state: unverified
                    >  to 185.15.59.158 via et-0/0/50.0
                    [BGP/170] 1w6d 06:11:59, MED 100, localpref 50
                      AS path: 64600 I, validation-state: unverified
                    >  to 10.80.0.2 via irb.321

TCAM usage is more than fine even on the QFX5100s:

cmooney@cloudsw1-c8-eqiad> show pfe route summary hw    

Slot 0

Unit: 0
Profile active: l2-profile-three
Type            Max       Used      Free      % free
----------------------------------------------------
IPv4 Host       147456    408       146716    99.50
IPv4 LPM        12288     376       11632     94.66
IPv4 Mcast      73728     0         73358     99.50

IPv6 Host       73728     166       73358     99.50
IPv6 LPM(< 64)  6144      140       5816      94.66
IPv6 LPM(> 64)  1024      173       851       83.11
IPv6 Mcast      36864     0         36679     99.50

Worth noting in the above that the /122 IPv6 prefixes from the K8s hosts are in their own memory partition, which is only allocated 20% of the memory reserved for regular IPv6 prefixes. But over 50% of our non-host v6 routes are of this type. We could potentially adjust the memory partition here, I've not looked at the options in depth. But we're still a bit away from capacity so no need to worry for now.

Change 971109 had a related patch set uploaded (by Cathal Mooney; author: Cathal Mooney):

[operations/homer/public@master] Export direct routes to Switches as well as OSPF & server BGP

https://gerrit.wikimedia.org/r/971109

Change 971109 merged by jenkins-bot:

[operations/homer/public@master] Export direct routes to Switches as well as OSPF & server BGP

https://gerrit.wikimedia.org/r/971109

cmooney raised the priority of this task from Low to Medium.

I had to revise the config slightly, it seems that local prefixes on OSPF-enabled interfaces are not included in "from protocol ospf". That meant that, for instance, local loopback interfaces weren't being exported. While generally not super-important it's a difference that could trip us up, and for codfw migration is relevant where servers on row-wide Vlans have BGP sessions to the CR loopbacks.

Also had to change the OSPF term "metric" command as it seems the 'set metric igp' only applies for BGP routes (based on next-hop), a different syntax is needed to copy cost from direct OSPF routes to BGP MED.

Change 971111 had a related patch set uploaded (by Cathal Mooney; author: Cathal Mooney):

[operations/homer/public@master] Fix syntax error in policy term

https://gerrit.wikimedia.org/r/971111

Change 971111 merged by jenkins-bot:

[operations/homer/public@master] Correctly set MED for OSPF routes

https://gerrit.wikimedia.org/r/971111

Routing now looks ok, for instance in esams to the loopbacks of each CR:

cmooney@asw1-bw27-esams> show route 185.15.59.128/32 

inet.0: 643 destinations, 1275 routes (643 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

185.15.59.128/32   *[BGP/170] 00:17:30, MED 0, localpref 100
                      AS path: 14907 I, validation-state: unverified
                    >  to 185.15.59.156 via et-0/0/48.0
                    [BGP/170] 00:00:24, MED 3, localpref 100
                      AS path: 14907 I, validation-state: unverified
                    >  to 185.15.59.158 via et-0/0/50.0

{master:0}
cmooney@asw1-bw27-esams> show route 185.15.59.129/32    

inet.0: 643 destinations, 1275 routes (643 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

185.15.59.129/32   *[BGP/170] 00:00:31, MED 0, localpref 100
                      AS path: 14907 I, validation-state: unverified
                    >  to 185.15.59.158 via et-0/0/50.0
                    [BGP/170] 00:17:37, MED 3, localpref 100
                      AS path: 14907 I, validation-state: unverified
                    >  to 185.15.59.156 via et-0/0/48.0

Icinga downtime and Alertmanager silence (ID=c5da2d0a-c4af-4f96-b651-e1b326898629) set by cmooney@cumin1001 for 2:00:00 on 34 host(s) and their services with reason: testing new bgp policy

lsw1-a[1-8]-codfw,lsw1-a[1-8]-codfw.mgmt,lsw1-b[2-8]-codfw,lsw1-b[2-8]-codfw.mgmt,ssw1-a[1,8]-codfw,ssw1-a[1,8]-codfw.mgmt

One other observation is that the MED setting does not optimize the outbound path where we are using EVPN.

One might hope that a LEAF switch, learning a given route from both SPINEs, would see the MED sent by the CR to each SPINE, and be able to pick the best Spine to forward traffic to for a given destination. But unfortunately the MED is not set on EVPN BGP routes at all. It's not even possible to set it via policy from what I can tell.

Not a big issue anyway. The SPINEs themselves always prefer their directly connected CR route over the one learnt from the other Spine, so we should never end up with "valley routing" (whereby a Spine prefers a route learnt from the other Spine, and sends traffic to it via an intermediate Leaf). All it means is that packets may get forwarded to CR1 which then get sent to CR2 before going over the WAN.

Announcing the specific routes is still of benefit, to ensure backup LVS is not used and to address the original issue of a CR that somehow gets disconnected from WAN.

One other observation is that the MED setting does not optimize the outbound path where we are using EVPN.

Juniper have this documented as a lacking feature on their site:

https://supportportal.juniper.net/s/article/EVPN-T5-route-attributes-that-inherited-from-the-original-routes

Without this function a packet from a LEAF going to one of our POPs might take a path from LEAF -> RANDOM_SPINE -> THAT_SPINES_CR -> EGRESS_CR.

If it propagated the MED the LEAF would pick the Spine connected to the CR that the outbound traffic will use, and path will be LEAF -> BEST_SPINE -> EGRESS_CR.

Which is one less hop and therefore advantageous. However the down side of this is, in the above example the Spine that is not "BEST" (RANDOM_SPINE on first line) would not automatically announce the route it learns from its CR. It prefers the IBGP EVPN route from the other Spine so doesn't tell the rest of the network about the fact it has a route to the destination direct from the CR also. This has the potential to cause momentary routing loops and packet loss after a link breaks while convergence takes place (we have to wait for the backup Spine to notice the route from the other Spine is gone, run the best path algorithm again, then start announcing the route its getting from CR to all the LEAFs).

In regular EBGP/IBGP (or with vendors who do propagate MED from unicast BGP to EVPN) the above issue during convergence can be mitigated by using advertise-external on the IBGP sessions. This command would cause the "backup Spine", in our case, to announce the inactive EBGP route from the CR even though it prefers the one it gets from the other Spine over IBGP. If the primary then dies all the LEAF devices already know about the backup option and can begin to use it as soon as the primary is withdrawn.

Anyway was looking into this today so figured worth documenting and adding to this task.