Page MenuHomePhabricator
Create Task
Maniphest T259593

Make eqord its own AS
Closed, ResolvedPublic
Actions

Description

eqord is part of the eqiad confederation, despite being closer to codfw in term of latency. This is a snowflake we can't get rid of, but could improve.

With T246721 we made eqord advertise only eqiad prefixes, and share the prefixes it learns only to eqiad (same confed).

Later on we made eqord advertise codfw prefixes as well (because closer to codfw).
To make eqord share the external prefixes it learned to codfw, without sharing the external prefixes learned from eqiad we have 2 options:

  • Use BGP policies
  • Separate eqord to its own confed

The 2nd option seems to be the cleaner on the overall design (there is no real reason it's part of eqiad).

Rough steps would be:

  • Disable external BGP sessions
  • Add the new AS# to confederation 14907 members [] (eg. 65020 to not impact future sites numbering).
  • Optionally, decide if we want to follow the physical links or keep the kinda full mesh we have -> keep the status quo
  • Convert eqiad-eqord iBGP sessions to eBGP (with an policy allowing to export its DFZ, but not importing any DFZ)
  • Renumber codfw/ulsfo neighbor AS
  • Enable external BGP sessions
  • Check that sessions are established and proper prefixes exchanged

Details

SubjectRepoBranchLines +/-
Change eqord ASN to 65020operations/homer/publicmaster+2 -2
Puppet: change eqord ASN to 65020operations/puppetproduction+3 -1
Customize query in gerrit

Related Objects

Event Timeline

ayounsi triaged this task as Medium priority.Aug 4 2020, 9:04 AM
ayounsi created this task.
Restricted Application added a project: SRE. · View Herald TranscriptAug 4 2020, 9:04 AM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
ayounsi renamed this task from Make eqord it's own AS to Make eqord its own AS.Aug 5 2020, 6:48 AM
ayounsi added a comment.Aug 19 2020, 1:13 PM

To be pushed, I'm only converting the existing sessions for now (not deleting/creating new ones).

cr2-eqord
[edit routing-options]
-   autonomous-system 65001;
+   autonomous-system 65020;
[edit routing-options confederation]
-   members [ 65001 65002 65003 65004 65005 ];
+   members [ 65001 65002 65003 65004 65005 65020 ];
[edit protocols bgp group Confed_codfw]
-    import iBGP_rpki;
-    export BGP_Wikimedia_no_dfz;
-    local-as 65001 no-prepend-global-as;
+    local-as 65020 no-prepend-global-as;
[edit protocols bgp group Confed_eqiad]
-    import iBGP_rpki;
+     multihop {
+         ttl 2;
+     }
-    type internal;
+    type external;
-    local-as 65001 no-prepend-global-as;
+    local-as 65020 no-prepend-global-as;
[edit protocols bgp group Confed_ulsfo]
-     local-as 65001 no-prepend-global-as;
+     local-as 65020 no-prepend-global-as;
[edit protocols bgp group Netflow]
-    peer-as 65001;
+    peer-as 65020;
-    local-as 65001 no-prepend-global-as;
+    local-as 65020 no-prepend-global-as;
cr1-eqiad
[edit routing-options confederation]
-   members [ 65001 65002 65003 65004 65005 ];
+   members [ 65001 65002 65003 65004 65005 65020 ];
[edit protocols bgp group Confed_eqiad]
-     neighbor 208.80.154.198;
[edit protocols bgp]
     group Netflow { ... }
+    group Confed_eqord {
+        type external;
+        multihop {
+            ttl 2;
+        }
+        local-address 208.80.154.196;
+        import iBGP_rpki;
+        family inet {
+            any;
+        }
+        family inet6 {
+            any;
+        }
+        export BGP_Wikimedia_no_dfz;
+        peer-as 65020;
+        local-as 65001 no-prepend-global-as;
+        neighbor 208.80.154.198;
+    }
cr2-eqiad
[edit routing-options confederation]
-   members [ 65001 65002 65003 65004 65005 ];
+   members [ 65001 65002 65003 65004 65005 65020 ];
[edit protocols bgp group Confed_eqiad]
-     neighbor 208.80.154.198;
[edit protocols bgp]
     group Netflow { ... }
+    group Confed_eqord {
+        type external;
+        multihop {
+            ttl 5;
+        }
+        local-address 208.80.154.197;
+        import iBGP_rpki;
+        family inet {
+            any;
+        }
+        family inet6 {
+            any;
+        }
+        export BGP_Wikimedia_no_dfz;
+        peer-as 65020;
+        local-as 65001 no-prepend-global-as;
+        neighbor 208.80.154.198;
+    }
cr2-codfw
[edit routing-options confederation]
-   members [ 65001 65002 65003 65004 65005 ];
+   members [ 65001 65002 65003 65004 65005 65020 ];
[edit protocols bgp group Confed_eqiad]
-     import iBGP_rpki;
-     neighbor 208.80.154.198;
[edit protocols bgp]
     group Netflow { ... }
+    group Confed_eqord {
+        type external;
+        local-address 208.80.153.193;
+        import iBGP_rpki;
+        family inet {
+            any;
+        }
+        family inet6 {
+            any;
+        }
+        export BGP_Wikimedia_no_dfz;
+        peer-as 65020;
+        local-as 65002 no-prepend-global-as;
+        neighbor 208.80.154.198;
+    }
cr3-ulsfo
[edit routing-options confederation]
-   members [ 65001 65002 65003 65004 65005 ];
+   members [ 65001 65002 65003 65004 65005 65020 ];
[edit protocols bgp group Confed_eqiad]
-     neighbor 208.80.154.198;
[edit protocols bgp]
     group Netflow { ... }
+    group Confed_eqord {
+        type external;
+        multihop {
+            ttl 5;
+        }
+        local-address 198.35.26.192;
+        family inet {
+            any;
+        }
+        family inet6 {
+            any;
+        }
+        export BGP_Wikimedia_pops;
+        peer-as 65020;
+        local-as 65004 no-prepend-global-as;
+        neighbor 208.80.154.198;
+    }
cr4-ulsfo
[edit routing-options confederation]
-   members [ 65001 65002 65003 65004 65005 ];
+   members [ 65001 65002 65003 65004 65005 65020 ];
[edit protocols bgp group Confed_eqiad]
-     neighbor 208.80.154.198;
[edit protocols bgp]
     group Netflow { ... }
+    group Confed_eqord {
+        type external;
+        multihop {
+            ttl 5;
+        }
+        local-address 198.35.26.193;
+        family inet {
+            any;
+        }
+        family inet6 {
+            any;
+        }
+        export BGP_Wikimedia_pops;
+        peer-as 65020;
+        local-as 65004 no-prepend-global-as;
+        neighbor 208.80.154.198;
+    }
ayounsi added a comment.Aug 20 2020, 7:39 AM

To clarify export/import policies:
From eqiad and codfw we export all WMF prefixes to eqord (and no DMZ), and apply the RPKI rules to the prefixes imported from eqord.
From ulsfo we export only the POPs prefixes to eqord, and export all WMF prefixes (no DMZ) from eqord.

gerritbot added a comment.Aug 25 2020, 6:39 AM

Change 622268 had a related patch set uploaded (by Ayounsi; owner: Ayounsi):
[operations/homer/public@master] Change eqord ASN to 65020

https://gerrit.wikimedia.org/r/622268

gerritbot added a project: Patch-For-Review.Aug 25 2020, 6:39 AM
gerritbot added a comment.Aug 25 2020, 6:43 AM

Change 622269 had a related patch set uploaded (by Ayounsi; owner: Ayounsi):
[operations/puppet@production] Puppet: change eqord ASN to 65020

https://gerrit.wikimedia.org/r/622269

gerritbot added a comment.Aug 25 2020, 6:45 AM

Change 622269 merged by Ayounsi:
[operations/puppet@production] Puppet: change eqord ASN to 65020

https://gerrit.wikimedia.org/r/622269

Stashbot added a comment.Aug 25 2020, 8:18 AM

Mentioned in SAL (#wikimedia-operations) [2020-08-25T08:18:11Z] <XioNoX> deactivate eqord peering/transit - T259593

Stashbot added a comment.Aug 25 2020, 8:19 AM

Mentioned in SAL (#wikimedia-operations) [2020-08-25T08:19:21Z] <XioNoX> reconfigure eqord to be AS65020 - T259593

Stashbot added a comment.Aug 25 2020, 8:50 AM

Mentioned in SAL (#wikimedia-operations) [2020-08-25T08:50:09Z] <XioNoX> re-activate eqord peering/transit - T259593

ayounsi updated the task description. (Show Details)Aug 25 2020, 8:57 AM
ayounsi closed this task as Resolved.Aug 25 2020, 9:01 AM

All done and checked that:
1/ internal prefixes are properly exchange in all direction (eg. ulsfo sees eqiad via eqord) even if not always the active path
2/ external prefixes learned in eqord are redistributed to eqiad and codfw and validation is properly applied
3/ external prefixes are not sent to ulsfo

gerritbot added a comment.Aug 25 2020, 9:07 AM

Change 622268 merged by jenkins-bot:
[operations/homer/public@master] Change eqord ASN to 65020

https://gerrit.wikimedia.org/r/622268

Maintenance_bot removed a project: Patch-For-Review.Aug 25 2020, 9:10 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL