Page MenuHomePhabricator

BGP: Investigate isolating codfw and eqiad
Closed, ResolvedPublic

Description

Currently eqiad and codfw are sharing all their external routing:

  • inbound: both codfw and eqiad IP prefixes (one /23 each) are advertised from both sites (and eqord) as a single /22
  • outbound: both sites exchange their full views

Which mean that traffic toward codfw VIPs can enter our network via eqiad (and the other way around).
And traffic exiting a codfw can be router to eqiad before an external transit/peering (and the other way around).

My guess is that is has been configured like that in the past so bandwidth from both sites was being utilized.

However this might not be the best option anymore, the main issue being with traffic engineering.

When esams is depooled, eqiad links tend to saturate. For that reason the traffic team used a different DNS geo-maps file to redirect some US regions to codfw instead of eqiad. But with the current routing strategy, even if the traffic goes to codfw's VIPs, it can still enters through eqiad, potentially saturating links.

The other reason to change this, is so both sites are more independent, splitting the /22 into two /23s would make a miss-configuration less likely to impact both sites. Similarly, not exchanging full views would make routing simpler (similar to our other sites) and convergence faster.

Details

Related Gerrit Patches:
operations/homer/public : masterShrink eqiad/eqord bgp_out to /23 and /48

Related Objects

Event Timeline

ayounsi created this task.Mon, Mar 2, 9:46 PM
Restricted Application added a project: Operations. · View Herald TranscriptMon, Mar 2, 9:46 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
akosiaris triaged this task as Medium priority.Thu, Mar 5, 8:49 AM
ema moved this task from Triage to Network on the Traffic board.Wed, Mar 11, 1:55 PM
CDanis added a subscriber: CDanis.Tue, Mar 17, 12:18 PM

digging more into it, codfw already advertise its own /23 (and v6 /48).

ayounsi added a comment.EditedThu, Mar 26, 11:21 AM

Next steps are:

  • Check ROAs - DONE
  • Advertise 208.80.154.0/23 (compete with CF) and 2620:0:861::/48 from eqiad/eqord
[edit routing-options aggregate]
     route 185.15.56.0/24 { ... }
+    route 208.80.154.0/23 policy BGP_aggregate_contributors;
[edit routing-options rib inet6.0 aggregate]
      route 2620:0:861:ed1a::/64 { ... }
+     route 2620:0:861::/48 policy BGP_aggregate_contributors;
[edit policy-options prefix-list bgp6-out]
+    2620:0:861::/48;

Note: 208.80.154.0/23 is already in bgp-out

  • Check that they are being advertised as expected (looking glass, RPKI status)
  • Cleanup 208.80.152.0/22 and 2620:0:860::/46
[edit routing-options rib inet6.0 aggregate]
-     route 2620:0:860::/46 policy BGP_aggregate_contributors;
[edit policy-options prefix-list bgp6-out]
-   2620:0:860::/46;
[edit routing-options aggregate]
-    route 208.80.152.0/22 policy BGP_aggregate_contributors;
[edit policy-options prefix-list bgp-out]
-    208.80.152.0/22;
  • In codfw/eqdfw set protocols bgp group Confed_eqiad export BGP_Wikimedia_no_dfz
  • In eqiad/eqord set protocols bgp group Confed_codfw export BGP_Wikimedia_no_dfz
  • Check that BGP prefixes are still being propagated between sites (eg. LVS VIPs)
  • Monitor traffic shift

Change 583579 had a related patch set uploaded (by Ayounsi; owner: Ayounsi):
[operations/homer/public@master] Shrink eqiad/eqord bgp_out to /23 and /48

https://gerrit.wikimedia.org/r/583579

Mentioned in SAL (#wikimedia-operations) [2020-03-26T15:40:01Z] <XioNoX> start advertising 2620:0:861::/48 from eqiad - T246721

  Prefix		  Nexthop	       MED     Lclpref    AS path
* 2620:0:860::/46         Self                                    I
* 2620:0:861::/48         Self                                    I

Advertised as expected.

Mentioned in SAL (#wikimedia-operations) [2020-03-26T15:49:18Z] <XioNoX> start advertising 208.80.154.0/23 from eqiad - T246721

Mentioned in SAL (#wikimedia-operations) [2020-03-26T16:12:53Z] <XioNoX> stop advertising 2620:0:860::/46 from eqiad - T246721

Mentioned in SAL (#wikimedia-operations) [2020-03-26T16:18:42Z] <XioNoX> stop advertising 208.80.152.0/22 from eqiad - T246721

Change 583579 merged by jenkins-bot:
[operations/homer/public@master] Shrink eqiad/eqord bgp_out to /23 and /48

https://gerrit.wikimedia.org/r/583579

Mentioned in SAL (#wikimedia-operations) [2020-03-26T16:34:11Z] <XioNoX> stop exchanging full BGP view between eqiad and codfw - T246721

This is all done.
Last step is to update IRRs, tracked in T235886.

ayounsi closed this task as Resolved.Fri, Mar 27, 2:59 PM
ayounsi claimed this task.

Done.