Management routers: use BGP instead of OSPF
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	ayounsi
	Nov 2 2021, 3:17 PM

Description

The management routers currently peer over OSPF with the core routers (prod) routers.

However our network would benefit migrating those sessions to BGP for a few reasons:

More filtering capabilities
Better debugging capability
Reduces our OSPF footprint (and thus blast radius and convergence time)
More aligned with our future network design

While the advantages OSPF brings here (faster failover/convergence) is less necessary for that part of the network.

The plan is to start with drmrs as it's not live yet, then progressively retrofit the other sites.

Details

Related Changes in Gerrit:

Subject	Repo	Branch	Lines +/-
Fix typo on ospf3	operations/homer/public	master	+1 -1
Remove OSFP from mr1-eqsin and cr2/3-eqsin	operations/homer/public	master	+2 -6
Adding BGP to mr1-eqsin, cr2/3-eqsin to replace OSPF	operations/homer/public	master	+9 -2
Add eqsin private IPV4 to prefix-list pops4	operations/homer/public	master	+1 -0
Add ulsfo private v4 range to prefix-list pops4	operations/homer/public	master	+1 -0
Add back replace ospf to mr.conf	operations/homer/public	master	+2 -0
Remove OSFP on mr1-ulsfo, cr3 and cr4 ulsfo	operations/homer/public	master	+0 -6
Add bgp to security zone production for mr	operations/homer/public	master	+1 -1
Allow bgp for security zone production on interface facing the cr3/cr4	operations/homer/public	master	+4 -0
Update peer-as on mr1-ulsfo to 14907	operations/homer/public	master	+2 -2
Add BGP on mr1-ulsfo and temporary remove replace ospf	operations/homer/public	master	+11 -2

Related Objects

Mentioned In: T421238: mr1-eqiad: move from OSPF to BGP
T320962: Decommission eqiad cage WiFi

Event Timeline

ayounsi created this task.Nov 2 2021, 3:17 PM

ayounsi triaged this task as Medium priority.

Restricted Application added a project: Infrastructure-Foundations. · View Herald TranscriptNov 2 2021, 3:17 PM

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

First draft here, v6 is not complete, but it's more to agree on the general direction, @cmooney, let me know what you think.

Once configured and confirmed working, I'll send the matching Homer patches.

mr1-drmrs

[edit security policies from-zone production to-zone junos-host]
      policy any--bfd { ... }
+     policy private--bgp {
+         match {
+             source-address wikimedia-private;
+             destination-address any;
+             application junos-bgp;
+         }
+         then {
+             permit;
+         }
+     }
      policy any--traceroute { ... }
[edit security zones security-zone production interfaces lo0.0 host-inbound-traffic protocols]
        ospf3 { ... }
+       bgp;
[edit security zones security-zone production interfaces ge-0/0/3.0 host-inbound-traffic protocols]
        ospf3 { ... }
+       bgp;
[edit security zones security-zone production interfaces ge-0/0/4.0 host-inbound-traffic protocols]
        ospf3 { ... }
+       bgp;
[edit interfaces ge-0/0/3 unit 0 family inet]
        address 185.15.58.142/31 { ... }
+       address 10.136.127.3/31;
[edit policy-options]
+   policy-statement BGP_Default {
+       term default {
+           from {
+               route-filter 0.0.0.0/0 exact;
+           }
+           then accept;
+       }
+       then reject;
+   }
+   policy-statement BGP_production {   
+       term direct {
+           from protocol direct;
+           then accept;
+       }
+       term loopback {
+           from interface lo0.0;
+           then accept;
+       }
+   }
[edit protocols]
+   bgp {
+       group Production4 {
+           type external;
+           import BGP_Default;
+           family inet {
+               unicast;
+           }
+           export BGP_production;
+           neighbor 10.136.127.2 {
+               description asw1-b12-drmrs;
+               peer-as 65006;
+           }
+       }
+       log-updown;
+   }
[edit routing-options]
+   autonomous-system 65000;

asw1-b12-drmrs

[edit interfaces ge-0/0/0 unit 0 family inet]
        address 185.15.58.143/31 { ... }
+       address 10.136.127.2/31;
[edit policy-options]
+   prefix-list management4 {
+       10.136.128.0/17;
+       185.15.58.130/32;
+   }
+   prefix-list management6 {
+       2a02:ec80:600:ffff::3/128;
+   }
[edit policy-options]
+   policy-statement BGP_Default {
+       term default {
+           from {
+               route-filter 0.0.0.0/0 exact;
+           }
+           then accept;
+       }
+       then reject;
+   }
+   policy-statement BGP_management {
+       term management4 {
+           from {
+               family inet6;
+               protocol bgp;
+               prefix-list-filter management4 exact;
+           }
+           then accept;
+       }
+       term management6 {
+           from {
+               protocol bgp;
+               prefix-list-filter management6 exact;
+           }
+           then accept;
+       }
+       then reject;
+   }
[edit protocols bgp]
     group Transit4 { ... }
+    group Management4 {
+        type external;
+        import BGP_management;         
+        family inet {
+            unicast;
+        }
+        export BGP_Default;
+        peer-as 65000;
+        neighbor 10.136.127.3 {
+            description mr1-drmrs;
+        }
+    }

Maintenance_bot added a project: SRE.Nov 2 2021, 3:45 PM

Looks good.

As discussed on irc I think the second term in "BGP_production" on mr1 isn't needed, although given how it works can hardly blame you for putting it there.

On the ASW "policy-statement BGP_management", the v4 term incorrectly says "family inet6" should be "family inet". The equivalent v6 term has no family explicitly set, but shouldn't really make a difference with the prefix match.

I think we need either a separate BGP group (Management6 / Production6) to do v6 peering and exchange the Telia IPv6 link address? Or we enable "family inet6" in the Management4 / Production4 groups, and do the v6 peering over v4 transport, but in that case the name of the group is kind of misleading?

I'm just realising you say v6 is not complete. So probably you are on top of this, but yep overall that looks good to me, going in the right direction. Do we have another peering to asw1-b13-drmrs from the mr?

ayounsi removed ayounsi as the assignee of this task.Aug 23 2022, 12:00 PM

ayounsi mentioned this in T320962: Decommission eqiad cage WiFi.Oct 19 2022, 9:19 AM

This is completed in drmrs, the same will be applied to the other sites when we bring L3 on the ToR switches as I don't think there is much value in changing the current cr<->mr setup.

In T294845#8758882, @ayounsi wrote:

This is completed in drmrs, the same will be applied to the other sites when we bring L3 on the ToR switches as I don't think there is much value in changing the current cr<->mr setup.

One thing I notice that does worry me here slightly, is that the MR participating in OSPF means it is providing a path between CRs at our L2 POPs. For instance from cr4-ulsfo we have this route for the cr3-ulsfo loopback:

cmooney@cr4-ulsfo> show route 198.35.26.192/32 exact                        

inet.0: 961077 destinations, 2091689 routes (960318 active, 0 holddown, 1084 hidden)
Restart Complete
+ = Active Route, - = Last Active, * = Both

198.35.26.192/32   *[OSPF/10] 3w6d 02:54:13, metric 2
                    >  to 198.35.26.196 via ae0.2
                       to 198.35.26.201 via et-0/0/1.402

While the preferred hop is via a LAG, there is only a single member port (40G et-0/0/0). If that port failed then the system would try to send any IBGP traffic that flowed between the routers via the MR link.

Using BGP instead of OSPF here would seem the way to resolve given that's the direction we've been going in anyway.

Re-assigning it to Papaul to do the change on ulsfo and eqsin.

It is a good training opportunity, and would remove moving parts the day we replace those sites' switches.

You can take example on how it's done in the other sites.

If that port failed then the system would try to send any IBGP traffic that flowed between the routers via the MR link

@cmooney indeed, that's one limitation of the "old" design the link between both core routers can't go down. That's why it's a lag (or now 2 distinct links). We set the metric between core routers and management router to 20000 also just in case.

In the process of Auditing Eqsin for mr1 migration from OSPF to BGP, I noticed some inconsistency.
1- There is no connection from msw2-eqsin to mr1-eqsin
2- cr1-eqsin is decommissioned but it is still connected to msw1-eqsin port 11 and 23
https://netbox.wikimedia.org/dcim/devices/744/interfaces/
3- cr2-eqsin and cr3-eqsin are connected to the same management switch msw1-eqsin
cr3 should be connected to msw1-eqsin and cr2 to msw2-eqsin
https://netbox.wikimedia.org/dcim/devices/744/interfaces/

Please see below the final diagram for ulsfo. Please review and let me know if all looks good so I can start the implementation .

mr diagram-ulsfo after.jpg (632×906 px, 59 KB)

Move scs-ulsfo connection from mr1-ulsfo

Right now scs-ulsfo is connected to mr1 on port ge-0/0/2, we need to disconnect it from mr port ge-0/0/2 and connect it to msw1-ulso and opening port

Move msw2-ulsfo connection from mr1 port ge-0/0/3 and connect it to port ge-0/0/2 on mr1-ulsfo

Note: The 2 lines below will be done only when we receive the new switches
-connect first asw1-b22 switch to port ge-0/0/3 on mr1-ulsfo (need 1000Base-T SEP Copper RJ-45 on the switch side https://www.fs.com/products/31389.html?attribute=109789&id=4617749)
-connect second asw1-b23 switch to prot ge-0/0/4 on mr1-ulsfo (need 1000Base-T SEP Copper RJ-45 on the switch side https://www.fs.com/products/31389.html?attribute=109789&id=4617749)

Note we need to keep in mind that the main goal here is to move the mgmt routers to use BGP instead of OSPF. It's fine to do some light recabling if it means we don't have to do it later on when we refresh the switches (and upgrade them to the new design). But no need for example to look at asw<->cr cabling.

For example ulsfo will happen soon enough with {T398504].

Note that until we have the new switches, the management routers will keep using the Xlink vlans (and the same IPs) to establish BGP to the core routers, like we do for OSPF. As the current switches are L2 only.

That said, the cabling in/out mr1-ulsfo in your diagram lgtm.

Understood

Change #1182185 had a related patch set uploaded (by Papaul; author: Papaul):

[operations/homer/public@master] Add bgg on mr1-ulsfo and temporary remove repalce ospf

https://gerrit.wikimedia.org/r/1182185

gerritbot added a project: Patch-For-Review.Aug 26 2025, 4:38 PM

Change #1182185 merged by Papaul:

[operations/homer/public@master] Add BGP on mr1-ulsfo and temporary remove replace ospf

https://gerrit.wikimedia.org/r/1182185

Diff on mr1-ulsfo

+   bgp {
+       group Production {
+           type external;
+           import BGP_Default;
+           export BGP_direct;
+           neighbor 198.35.26.198 {
+               description cr3-ulsfo;
+               family inet {
+                   unicast;
+               }
+               peer-as 65004;
+           }
+           neighbor 2620:0:863:fe05::1 {
+               description cr3-ulsfo;
+               family inet6 {
+                   unicast;
+               }
+               peer-as 65004;
+           }
+           neighbor 198.35.26.200 {
+               description cr4-ulsfo;
+               family inet {
+                   unicast;
+               }
+               peer-as 65004;
+           }
+           neighbor 2620:0:863:fe06::1 {
+               description cr4-ulsfo;
+               family inet6 {
+                   unicast;
+               }
+               peer-as 65004;
+           }
+       }
+   }

Diff on cr3-ulsdo

     group Anycast6 { ... }
+    group Management {
+        type external;
+        export BGP_Default;
+        peer-as 65000;
+        neighbor 198.35.26.199 {
+            description mr1-ulsfo;
+            family inet {
+                unicast;
+            }
+        }
+        neighbor 2620:0:863:fe05::2 {
+            description mr1-ulsfo;
+            family inet6 {
+                unicast;
+            }
+        }
+    }

Diff on cr4-ulsfo

[edit protocols bgp]
     group Anycast6 { ... }
+    group Management {
+        type external;
+        export BGP_Default;
+        peer-as 65000;
+        neighbor 198.35.26.201 {
+            description mr1-ulsfo;
+            family inet {
+                unicast;
+            }
+        }
+        neighbor 2620:0:863:fe06::2 {
+            description mr1-ulsfo;
+            family inet6 {
+                unicast;
+            }
+        }
+    }

Maintenance_bot removed a project: Patch-For-Review.Aug 27 2025, 1:31 PM

Change #1182608 had a related patch set uploaded (by Papaul; author: Papaul):

[operations/homer/public@master] Update peer-as on mr1-ulsfo to 14907

https://gerrit.wikimedia.org/r/1182608

gerritbot added a project: Patch-For-Review.Aug 27 2025, 3:50 PM

Change #1182608 merged by Papaul:

[operations/homer/public@master] Update peer-as on mr1-ulsfo to 14907

https://gerrit.wikimedia.org/r/1182608

Change #1182612 had a related patch set uploaded (by Papaul; author: Papaul):

[operations/homer/public@master] Allow bgp for security zone production on interface facing the cr3/cr4

https://gerrit.wikimedia.org/r/1182612

Change #1182612 merged by Papaul:

[operations/homer/public@master] Allow bgp for security zone production on interface facing the cr3/cr4

https://gerrit.wikimedia.org/r/1182612

Maintenance_bot removed a project: Patch-For-Review.Aug 27 2025, 4:31 PM

Change #1182653 had a related patch set uploaded (by Papaul; author: Papaul):

[operations/homer/public@master] Add bgp to security zone production for mr

https://gerrit.wikimedia.org/r/1182653

gerritbot added a project: Patch-For-Review.Aug 27 2025, 8:30 PM

Change #1182653 merged by Papaul:

[operations/homer/public@master] Add bgp to security zone production for mr

https://gerrit.wikimedia.org/r/1182653

BGP is up and running between mr1-ulsfo and both cr3 and cr4

Peer                     AS      InPkt     OutPkt    OutQ   Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped...
198.35.26.198         14907       1102       1193       0       0     8:56:46 Establ
  inet.0: 0/1/1/0                       
198.35.26.200         14907          6          6       0       0        1:48 Establ
  inet.0: 0/1/1/0                       
2620:0:863:fe05::1       14907       1095       1185       0       0     8:53:27 Establ
  inet6.0: 0/1/1/0                      
2620:0:863:fe06::1       14907          5          6       0       0        1:48 Establ
  inet6.0: 0/0/0/0

mr1-ulsfo> show route protocol ospf   

inet.0: 206 destinations, 208 routes (206 active, 0 holddown, 0 hidden)

mr1-ulsfo> show route protocol bgp   

inet.0: 206 destinations, 208 routes (206 active, 0 holddown, 0 hidden)

Maintenance_bot removed a project: Patch-For-Review.Aug 27 2025, 11:31 PM

Good job!!

Next step is to remove OSPF, feel free to send a patch in that regard. It can restore the previously removed replace: ospf {} as well as remove those lines from the config: https://github.com/wikimedia/operations-homer-public/blob/master/config/common.yaml#L179C1-L184C20

- cr3-ulsfo.wikimedia.org: et-0/0/1.401
  mr1-ulsfo.wikimedia.org: ge-0/0/4.401
  metric: 20000
- cr4-ulsfo.wikimedia.org: et-0/0/1.402
  mr1-ulsfo.wikimedia.org: ge-0/0/4.402
  metric: 20000

Once approved:

make sure we can reach the mgmt router from OOB (mr1-ulsfo.oob.wikimedia.org)
downtime the device
manually disable OSPF (using commit confirmed)
check that everything is still working as expected (mgmt network and mr1's loopback still reachable)
Deploy the patch to the cr and mr1

Then you can look at doing the same for eqsin :)

@ayounsi thank you I will get back on this next week since i am off tomorrow and Monday is a U.S holiday.

Change #1184202 had a related patch set uploaded (by Papaul; author: Papaul):

[operations/homer/public@master] Remove OSFP on mr1-ulsfo, cr3 and cr4 ulsfo

https://gerrit.wikimedia.org/r/1184202

gerritbot added a project: Patch-For-Review.Sep 3 2025, 5:06 AM

Change #1184204 had a related patch set uploaded (by Papaul; author: Papaul):

[operations/homer/public@master] Add back replace ospf to mr.conf

https://gerrit.wikimedia.org/r/1184204

Change #1184202 merged by Papaul:

[operations/homer/public@master] Remove OSFP on mr1-ulsfo, cr3 and cr4 ulsfo

https://gerrit.wikimedia.org/r/1184202

Change #1184204 merged by Papaul:

[operations/homer/public@master] Add back replace ospf to mr.conf

https://gerrit.wikimedia.org/r/1184204

Maintenance_bot removed a project: Patch-For-Review.Sep 3 2025, 3:31 PM

manually disable OSPF (using commit confirmed) make the mgmt goes down when done on mr1 or cr3/cr4 . But mr1-ulsfo.oob.wikimedia.org and mr1 loopback still reachable. I need to investigate more to see why. When ospf disable here are the routes to 10.128.128.0./17 (ulsfo mgmt)

cr3-ulsfo> show route 10.128.128.0/17 

inet.0: 990018 destinations, 2935493 routes (989924 active, 1 holddown, 379 hidden)
Restart Complete
+ = Active Route, - = Last Active, * = Both

10.128.128.0/17    *[BGP/170] 1w0d 01:46:22, localpref 100
                      AS path: 65000 I, validation-state: unverified
                    >  to 198.35.26.199 via et-0/0/1.401
                    [BGP/170] 00:11:34, localpref 100, from 198.35.26.193
                      AS path: 65000 I, validation-state: unverified
                    >  to 198.35.26.213 via et-0/0/0.0
                       to 198.35.26.197 via ae0.2

Icinga downtime and Alertmanager silence (ID=829d4d0b-c9d0-4961-b07b-d12e8f1ac430) set by pt1979@cumin2002 for 2:00:00 on 1 host(s) and their services with reason: Bgp testing

mr1-ulsfo

Change #1184816 had a related patch set uploaded (by Ayounsi; author: Ayounsi):

[operations/homer/public@master] Add ulsfo private v4 range to prefix-list pops4

https://gerrit.wikimedia.org/r/1184816

gerritbot added a project: Patch-For-Review.Sep 4 2025, 3:02 PM

Change #1184816 merged by Papaul:

[operations/homer/public@master] Add ulsfo private v4 range to prefix-list pops4

https://gerrit.wikimedia.org/r/1184816

mr1-ulsfo is now running BGP . All OSPF entries on mr1-ulsfo, cr3-ulsfo and cr4-ulsfo for the management network removed.

Maintenance_bot removed a project: Patch-For-Review.Sep 4 2025, 11:31 PM

Change #1185112 had a related patch set uploaded (by Papaul; author: Papaul):

[operations/homer/public@master] Adding BGP to mr1-eqsin, cr2/3-eqsin to replace OSPF

https://gerrit.wikimedia.org/r/1185112

gerritbot added a project: Patch-For-Review.Sep 5 2025, 3:06 PM

Change #1185115 had a related patch set uploaded (by Papaul; author: Papaul):

[operations/homer/public@master] Add eqsin privare IPV4 to prefix-list pops4

https://gerrit.wikimedia.org/r/1185115

Change #1185112 merged by jenkins-bot:

[operations/homer/public@master] Adding BGP to mr1-eqsin, cr2/3-eqsin to replace OSPF

https://gerrit.wikimedia.org/r/1185112

Change #1185115 merged by jenkins-bot:

[operations/homer/public@master] Add eqsin private IPV4 to prefix-list pops4

https://gerrit.wikimedia.org/r/1185115

Maintenance_bot removed a project: Patch-For-Review.Sep 8 2025, 1:31 PM

BGP is up on mr1-eqsin cr2/3-eqsin

mr1-eqsin# run show route protocol ospf

inet.0: 198 destinations, 200 routes (198 active, 0 holddown, 0 hidden)
Restart Complete

mr1-eqsin# run show route protocol bgp

inet.0: 198 destinations, 200 routes (198 active, 0 holddown, 0 hidden)
Restart Complete
+ = Active Route, - = Last

mr1-eqsin# run show bgp summary
Threading mode: BGP I/O
Default eBGP mode: advertise - accept, receive - accept
Groups: 1 Peers: 4 Down peers: 0
Table          Tot Paths  Act Paths Suppressed    History Damp State    Pending
inet.0
                       2          0          0          0          0          0
inet6.0
                       1          0          0          0          0          0
Peer                     AS      InPkt     OutPkt    OutQ   Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped...
103.102.166.132       14907          6          6       0       0        1:41 Establ
  inet.0: 0/1/1/0
103.102.166.142       14907         12         12       0       0        4:26 Establ
  inet.0: 0/1/1/0
2001:df2:e500:fe03::1       14907          5          6       0       0        1:41 Establ
  inet6.0: 0/0/0/0
2001:df2:e500:fe04::1       14907         11         12       0       0        4:25 Establ
  inet6.0: 0/1/1/0

I will remove OSFP later

ayounsi moved this task from Backlog to This quarter on the netops board.Sep 9 2025, 9:52 AM

Change #1186682 had a related patch set uploaded (by Papaul; author: Papaul):

[operations/homer/public@master] Remove OSFP from mr1-eqsin and cr2/3-eqsin

https://gerrit.wikimedia.org/r/1186682

gerritbot added a project: Patch-For-Review.Sep 10 2025, 3:56 AM

Icinga downtime and Alertmanager silence (ID=eb9214cb-2708-4519-b3c6-38d4f0f7cf7d) set by pt1979@cumin1002 for 1:00:00 on 1 host(s) and their services with reason: router upgrade