Page MenuHomePhabricator

Standardize cross confederation BGP policies
Closed, ResolvedPublic0 Estimated Story Points


Between sites (confederations) we currently have those 3 options:

  1. Export full views (no export filters, used between eqiad/codfw)
  2. BGP_Wikimedia_no_dfz Export Wikimedia private/public prefixes learned via BGP (eg. used on eqiad->ulsfo, codfw->ulsfo, ulsfo->eqsin)
  3. BGP_Wikimedia_own_space Export public prefixes (same as to DFZ) + customers prefixes (if any) (eg. used on eqiad->esams, esams->eqiad, ulsfo->eqiad)

My proposal here is to add the customers term to BGP_Wikimedia_no_dfz and replace all export BGP_Wikimedia_own_space with export BGP_Wikimedia_no_dfz.

This has the following advantages:

  • Standardized and less complex (one less) routing policies
  • Anycast VIPs (in the private space) will be routed across all sites (eg. if esams rec-dns servers are down, esams clients will route to eqiad)

I can't see any downsides, but I also don't have the historical reasons on why it has been configured that way.

Changes are:

all routers
[edit policy-options policy-statement BGP_Wikimedia_no_dfz]
     term public6 { ... }
+    term customers {
+        from community PEER_CUSTOMER;
+        then accept;
+    }
-    term reject_other {
-        then reject;
-    }
[edit policy-options policy-statement BGP_Wikimedia_no_dfz]
+    then reject;
[edit policy-options]
-   policy-statement BGP_Wikimedia_own_space {
-       term local4 {
-           from policy BGP_outfilter;
-           then accept;
-       }
-       term local6 {
-           from policy BGP6_outfilter;
-           then accept;
-       }
-       then reject;
-   }

(this also moves the reject from a term to a default)

router specific (eg. eqiad)
[edit protocols bgp group Confed_esams]
-    export BGP_Wikimedia_own_space;
+    export BGP_Wikimedia_no_dfz;

To be deployed progressively site by site.

Event Timeline

ayounsi created this task.
Restricted Application added a subscriber: Aklapper. · View Herald Transcript

Mentioned in SAL (#wikimedia-operations) [2019-07-25T21:46:22Z] <XioNoX> apply export BGP_Wikimedia_no_dfz to eqiad's Confed_esams - T227808

Confirmed working as expected, eg. esams still show the customer prefixes, plus now BGP advertised prefixes (LVS/Anycast).
Will let it sit before rolling out to all sites.

This is done and pushed to all the sites.

ayounsi reopened this task as Open.EditedAug 15 2019, 10:42 PM

Reopening as the cleanup above is only part of the solution. It was made with the idea that it would be ok for all sites to route to any other site, while as explained by Brandon in T228190#5414366 it's better to influence that routing.

As we also advertise internal and other non BGP prefixes (including statics) via OSPF we could technically do with only advertising customers between sites. But that's outside the scope here and for another discussion.

CORE <-> CORE full view (already there)
CORE -> POP and POP -> POP: core BGP (LVS + k8s + frack) + pops BGP (LVS) + customers (office) + anycast (already there)
This is so for example eqsin knows how to reach the cores via ulsfo if the ulsfo-codfw link goes down
POP -> CORE: pops BGP (LVS) + customers (to be created)

To do that we can create a prefix-list with all POPs v4 and v6 prefixes (similar to wikimedia4/6).
And allow only those prefixes (orlonger, to include VIPs/32s and /128) to be advertised from any POP to any CORE.

[edit policy-options]
+   prefix-list pops4 {
+   }
+   prefix-list pops6 {
+       2001:df2:e500::/48;
+       2620:0:862::/48;
+       2620:0:863::/48;
+       2a02:ec80::/32;
+   }
[edit policy-options]
+   policy-statement BGP_Wikimedia_pops {
+       term pops4 {
+           from {
+               protocol bgp;
+               prefix-list-filter pops4 orlonger;
+           }
+           then accept;
+       }
+       term pops6 {
+           from {
+               protocol bgp;
+               prefix-list-filter pops6 orlonger;
+           }
+           then accept;
+       }
+       term customers {
+           from community PEER_CUSTOMER;
+           then accept;
+       }
+       then reject;
+   }
[edit protocols bgp group Confed_eqiad] # If any
-    export BGP_Wikimedia_no_dfz;
+    export BGP_Wikimedia_pops;
[edit protocols bgp group Confed_codfw] # If any
-    export BGP_Wikimedia_no_dfz;
+    export BGP_Wikimedia_pops;

Mentioned in SAL (#wikimedia-operations) [2019-08-20T19:04:58Z] <XioNoX> push BGP_Wikimedia_pops to cr3-ulsfo - T227808

Updated the change to no not include the local aggregates to keep the level of changes to a minimum. I do think that's something we would want to add down the road though.
Confirmed that cr3-ulsfo advertise the same prefixes to cr1-codfw minus the follow 3: (anycast) (eqiad, used to be learned via a longer AS path: eqiad->ulsfo->codfw) (eqiad, same as above)

Mentioned in SAL (#wikimedia-operations) [2019-08-20T19:25:14Z] <XioNoX> push BGP_Wikimedia_pops to cr4-ulsfo - T227808

Mentioned in SAL (#wikimedia-operations) [2019-08-20T19:28:45Z] <XioNoX> push BGP_Wikimedia_pops to eqsin - T227808

Mentioned in SAL (#wikimedia-operations) [2019-08-20T20:49:12Z] <XioNoX> push BGP_Wikimedia_pops to ams - T227808