It was discovered there is a gap in the logic for originating aggregate routes into BGP on our core routers. This was discovered on the back of a query from Bell Canada asking why we were not announcing IPv4 prefix 208.80.154.0/23 to them from eqord, but only from eqiad.
The current configuration is set up to create certain aggregate routes if any longer prefixes from within them are present in the BGP RIB. In our case it should mean that the presence of routes originated from LVS at remote sites will trigger the creation of the configured aggregates. Looking at the above prefix the relevant config on cr2-eqord is like this:
```
set routing-options aggregate route 208.80.154.0/23 policy BGP_from_LVS
set policy-options policy-statement BGP_from_LVS term BGP_core_and_local_LVS from protocol bgp
set policy-options policy-statement BGP_from_LVS term BGP_core_and_local_LVS from as-path core_and_local_LVS
set policy-options policy-statement BGP_from_LVS term BGP_core_and_local_LVS then accept
set policy-options policy-statement BGP_from_LVS then reject
set policy-options as-path core_and_local_LVS "^(65002|65001)? 64600.*"
```
The one constraint on just creating the aggregate is the as-path regex in the last line. All LVS instances use ASN 64600, so the regex basically says "routes originated from LVS ASN, with either AS65002 (codfw) or AS65001 (eqiad) in the path, but not both". The intent of the configuration is to create the aggregate route only if it is being learnt directly from the site where it is being used (codfw or eqiad), but avoid doing so if it is being learnt from another site. The idea is we don't want to announce a route in Chicago for an Ashburn prefix if our network wants to get there via Dallas, for instance if our transport links from Chicago to Ashburn are down.
A problem can occur because of the interrelationship of BGP and OSPF, however, and how the BGP best-path algorithm works when confederations are in use. Consider the current BGP best-path for prefix 208.80.154.224/32 (origingated by lvs1013 in eqiad), on cr2-eqord:
```
cmooney@cr2-eqord> show route protocol bgp 208.80.154.224/32 terse
inet.0: 833225 destinations, 2049355 routes (832285 active, 1 holddown, 1511 hidden)
Restart Complete
+ = Active Route, - = Last Active, * = Both
A V Destination P Prf Metric 1 Metric 2 Next hop AS path
* ? 208.80.154.224/32 B 170 100 0 (65002 65001) 64600 I
unverified >208.80.154.208
? B 170 100 0 (65001) 64600 I
unverified >208.80.154.208
? B 170 100 0 (65001) 64600 I
unverified >208.80.154.208
```
Local preference and MED are the same on all 3 of these routes. The router has ended up using the router-id attribute from each as tie-break, resulting in the one learnt from codfw being selected. Notably the fact that there are 2 sub-as's (65002 65001) in the path for this route, as opposed to only 1 (65001) on those learnt directly from eqiad, is not considered when selecting the best path. This is normal behaviour with BGP confederations, the sub-as path is thought of as "1 hop" regardless of how many ASNs are between the brackets.
For our normal routing to this prefix it is not in any way an issue that BGP has selected the route that propagated through codfw to get to eqord. Whether learnt from codfw, or direct from eqiad, the next-hop in the BGP message is the IP of the originating LVS server in eqiad, 10.64.1.13:
```
cmooney@cr2-eqord> show route protocol bgp 208.80.154.224/32 detail | match "Source|Protocol Next Hop"
Source: 208.80.153.193
Protocol next hop: 10.64.1.13
Source: 208.80.154.196
Protocol next hop: 10.64.1.13
Source: 208.80.154.197
Protocol next hop: 10.64.1.13
```
So regardless of which BGP route is selected, the same (indirect) next-hop IP is going to be used, and in normal circumstance traffic will route directly to eqiad due to lower IGP cost:
```
cmooney@cr2-eqord> show route 10.64.1.13
inet.0: 833306 destinations, 2049530 routes (832369 active, 0 holddown, 1493 hidden)
Restart Complete
+ = Active Route, - = Last Active, * = Both
10.64.0.0/22 *[OSPF/10] 1w5d 21:47:37, metric 242
> to 208.80.154.208 via xe-0/1/5.0
```
Going back to the BGP policy on the aggregate route, however, there is an issue. Despite the traffic still routing directly to eqiad, the fact BGP has selected the route telling it that from codfw means the as-path regex isn't matched. The regex only permits the BGP routes learnt directly from eqiad, neither of which aren't the selected best path:
```
cmooney@cr2-eqord> show route protocol bgp 208.80.154.224/32 aspath-regex "^(65002|65001)? 64600.*"
inet.0: 833377 destinations, 2049658 routes (832435 active, 0 holddown, 1498 hidden)
Restart Complete
+ = Active Route, - = Last Active, * = Both
208.80.154.224/32 [BGP/170] 1w5d 21:57:29, MED 0, localpref 100, from 208.80.154.196
AS path: (65001) 64600 I, validation-state: unverified
> to 208.80.154.208 via xe-0/1/5.0
[BGP/170] 1w5d 21:57:29, MED 0, localpref 100, from 208.80.154.197
AS path: (65001) 64600 I, validation-state: unverified
> to 208.80.154.208 via xe-0/1/5.0
```
How to overcome this is a tricky question. I don't believe an AS-path filter on the aggregate can be used as:
- We know that route with longer sub-as path can be selected as best.
- Due to use of confederations the next-hop is preserved.
- The outbound interface for that next-hop is not based on anything in BGP.
- AS-Path on the selected BGP route thus doesn't tell us anything about the satus of eqord -> eqiad link(s).
In terms of a solution there are two broad approaches:
1. Change the BGP policy so that the route with the longer sub-AS path will not be selected, keeping policy on the aggregate the same.
2. Change the policy on the aggregate so it will be created no matter what (sub) AS-path is on the BGP best route, but use some other criteria to not create it if the IGP route isn't optimal.