Page MenuHomePhabricator

Unify loopback filters between CR routers and L3 switches
Closed, ResolvedPublic

Description

The requirements for filtering on the loopback, i.e. filtering traffic to the system Routing Engine / JunOS itself, is essentially the same between our layer-3 switches (i.e. drmrs QFX5120 devices) and core routers (typically MX480 or MX204s).

So it is realistic for us to use the same Capirca policy definition to build the 'loopback4' and 'loopback6' filters for all of them. There are certain terms that aren't needed in both cases, but won't cause any issue if they are configured and get no matches (i.e. there is no VRRP on L3 switches, EVPN devices need to include overlay BGP sessions the others don't have).

Creating this task with the aim of:

  1. Adjusting the capirca policy for cr-loopbacks, so it will build the filter in a way that is compatible / has same effect as existing (manually configured) filters on drmrs asw's.
  2. Further modifying the policy to include the additional elements needed for EVPN lsw/ssw.

Event Timeline

cmooney triaged this task as Medium priority.Mar 23 2022, 6:24 PM
cmooney created this task.
Restricted Application added a subscriber: Aklapper. · View Herald Transcript

Change 773299 had a related patch set uploaded (by Cathal Mooney; author: Cathal Mooney):

[operations/homer/public@master] Modify cr-loopback Capirca definition to make it compatible with QFX

https://gerrit.wikimedia.org/r/773299

The diff in the above patch is relatively straightforward on the CRs, basically the same as you can see looking at the policy changes:

Changes for 1 devices: ['cr1-eqiad.wikimedia.org']

[edit firewall family inet filter loopback4]
       term allow_ssh4 { ... }
+      term allow_bgp_src {
+          from {
+              source-prefix-list {
+                  bgp-sessions;
+              }
+              protocol tcp;
+              source-port 179;
+          }
+          then accept;
+      }
+      term allow_bgp_dst {
+          from {
+              source-prefix-list {
+                  bgp-sessions;
+              }
+              protocol tcp;
+              destination-port 179;
+          }
+          then accept;
+      }
       term allow_ospf { ... }
[edit firewall family inet filter loopback4 term allow_dhcp_request4 from]
-       destination-port 67;
+       destination-port 67-68;
[edit firewall family inet filter loopback4 term allow_dhcp_reply4 from]
-       destination-port 67;
+       destination-port 67-68;
[edit firewall family inet filter loopback4]
-      term allow_bgp {
-          from {
-              source-prefix-list {
-                  bgp-sessions;
-              }
-              protocol tcp;
-              port 179;
-          }
-          then accept;
-      }
[edit firewall family inet6 filter loopback6]
       term allow_ssh6 { ... }
+      term allow_bgp_src {
+          from {
+              source-prefix-list {
+                  bgp-sessions;
+              }
+              next-header tcp;
+              source-port 179;
+          }
+          then accept;
+      }
+      term allow_bgp_dst {
+          from {
+              source-prefix-list {
+                  bgp-sessions;
+              }
+              next-header tcp;
+              destination-port 179;
+          }
+          then accept;
+      }
       term allow_ospf { ... }
[edit firewall family inet6 filter loopback6]
-      term allow_bgp {
-          from {
-              source-prefix-list {
-                  bgp-sessions;
-              }
-              next-header tcp;
-              port 179;
-          }
-          then accept;
-      }

---------------
INFO:homer:Homer run completed successfully on 1 devices: ['cr1-eqiad.wikimedia.org']

In drmrs the changes are more involved, and it's a little harder to read the homer diff. Ultimately they are only small and shouldn't change how anything works.

Both protocols:

  • The order of some terms have changed, but contents are unchanged. I've stuck with order used on CRs.
  • Names of some will change in drmrs: "allow_ospf4" is now "allow_ospf", same for allow_bfd and allow_dns.
    • I stuck with CR naming

loopback4:

  • BGP src/dst term, this is needed to allow BGP to establish in either direction.
    • The 'return-tcp' term largely covers this for drmrs, but it matches production4 prefix list
    • It thus won't cover external BGP peers so the two terms are needed on CRs if moving away from "port"
  • Port 67 and 68 are included in the destination ports for both DHCP terms
    • CRs only had port 67 for destination port on these before
    • Capirca is defaulting to making it a "range", i.e. 67-68, whereas drmrs is currently configured with a list [ 67 68 ]

loopback6

  • Split BGP into src/dst terms
  • Potential typo fixed - SNMP term changing from "proto tcp" to "proto udp".
  • Allow NTP term in drmrs is currently after the deny-all, patch changes order to match CRs.
  • allow-junos-upgrades term added in drmrs
  • deny_other changed name to deny_all in drmrs

To clarify the 'port' isn't an option on QFX even for UDP, although it allows you to define a term with that.

So I've changed the 'port' to 'destination-port' for NTP and BFD. This results in a bigger diff on the CRs

Changes for 1 devices: ['cr1-eqiad.wikimedia.org']

[edit firewall family inet filter loopback4]
       term allow_ssh4 { ... }
+      term allow_bgp_src {
+          from {
+              source-prefix-list {
+                  bgp-sessions;
+              }
+              protocol tcp;
+              source-port 179;
+          }
+          then accept;
+      }
+      term allow_bgp_dst {
+          from {
+              source-prefix-list {
+                  bgp-sessions;
+              }
+              protocol tcp;
+              destination-port 179;
+          }
+          then accept;
+      }
       term allow_ospf { ... }
[edit firewall family inet filter loopback4 term allow_ntp4 from]
-       source-prefix-list {
-           production4;
-       }
-       protocol udp;
-       port 123;
+       source-prefix-list {
+           production4;
+       }
+       protocol udp;
+       destination-port 123;
[edit firewall family inet filter loopback4 term allow_dhcp_request4 from]
-       destination-port 67;
+       destination-port 67-68;
[edit firewall family inet filter loopback4 term allow_dhcp_reply4 from]
-       destination-port 67;
+       destination-port 67-68;
[edit firewall family inet filter loopback4 term allow_bfd from]
-       protocol udp;
-       port 3784-3785;
+       protocol udp;
+       destination-port 3784-3785;
[edit firewall family inet filter loopback4 term allow_mbfd from]
-       source-prefix-list {
-           bgp-sessions;
-       }
-       protocol udp;
-       port 4784;
+       source-prefix-list {
+           bgp-sessions;
+       }
+       protocol udp;
+       destination-port 4784;
[edit firewall family inet filter loopback4]
-      term allow_bgp {
-          from {
-              source-prefix-list {
-                  bgp-sessions;
-              }
-              protocol tcp;
-              port 179;
-          }
-          then accept;
-      }
[edit firewall family inet6 filter loopback6]
       term allow_ssh6 { ... }
+      term allow_bgp_src {
+          from {
+              source-prefix-list {
+                  bgp-sessions;
+              }
+              next-header tcp;
+              source-port 179;
+          }
+          then accept;
+      }
+      term allow_bgp_dst {
+          from {
+              source-prefix-list {
+                  bgp-sessions;
+              }
+              next-header tcp;
+              destination-port 179;
+          }
+          then accept;
+      }
       term allow_ospf { ... }
[edit firewall family inet6 filter loopback6 term allow_ntp6 from]
-       source-prefix-list {
-           production6;
-       }
-       next-header udp;
-       port 123;
+       source-prefix-list {
+           production6;
+       }
+       next-header udp;
+       destination-port 123;
[edit firewall family inet6 filter loopback6 term allow_bfd from]
-       next-header udp;
-       port 3784-3785;
+       next-header udp;
+       destination-port 3784-3785;
[edit firewall family inet6 filter loopback6 term allow_mbfd from]
-       source-prefix-list {
-           bgp-sessions;
-       }
-       next-header udp;
-       port 4784;
+       source-prefix-list {
+           bgp-sessions;
+       }
+       next-header udp;
+       destination-port 4784;
[edit firewall family inet6 filter loopback6]
-      term allow_bgp {
-          from {
-              source-prefix-list {
-                  bgp-sessions;
-              }
-              next-header tcp;
-              port 179;
-          }
-          then accept;
-      }

---------------
INFO:homer:Homer run completed successfully on 1 devices: ['cr1-eqiad.wikimedia.org']

Ultimately this should be fine. NTP is src+dst 123, BFD is always sent to destination 3784-3785/4784.

Change 773299 merged by jenkins-bot:

[operations/homer/public@master] Modify cr-loopback Capirca definition to make it compatible with QFX

https://gerrit.wikimedia.org/r/773299

Mentioned in SAL (#wikimedia-operations) [2022-04-11T11:34:56Z] <topranks> Adjust loopback filter on cr3-ulsfo to align with L3 switch config. T304553.

Mentioned in SAL (#wikimedia-operations) [2022-04-11T11:46:48Z] <topranks> Adjust loopback filter on asw1-b12-drmrs to align with CR router config. T304553.

Mentioned in SAL (#wikimedia-operations) [2022-04-12T12:51:38Z] <topranks> modify loopback filter on cr3-ulsfo to add terms needed in evpn context T304553

Mentioned in SAL (#wikimedia-operations) [2022-04-12T13:38:08Z] <topranks> Adding loopback4 filter to lo0.0 interface ingress lsw1-e1-eqiad T304553

Change 779510 had a related patch set uploaded (by Cathal Mooney; author: Cathal Mooney):

[operations/homer/public@master] Modify CR loopback filter and add VRF-specific filter for switches

https://gerrit.wikimedia.org/r/779510

Change 779510 merged by jenkins-bot:

[operations/homer/public@master] Modify CR loopback filter and add VRF-specific filter for switches

https://gerrit.wikimedia.org/r/779510

On the EVPN devices filtering needs to be defined on each 'unit' of the loopback interface, i.e. the default one "lo0.0" in the global routing instance, as well as "lo0.1" and any others that are defined for other routing instances.

We *could* apply the same filter to the lo0.1 interface for our EVPN switches, which had been my plan, but when doing this I hit an issue:

Message from syslogd@lsw1-e1-eqiad at Apr 12 14:44:13  ...
lsw1-e1-eqiad fpc0 DFWE ERROR DFW: Cannot program filter "loopback6-lo0.1-i" (type IRACL) - TCAM has 16 free entries and the filter requires 164 free entries 

Message from syslogd@lsw1-e1-eqiad at Apr 12 14:44:13  ...
lsw1-e1-eqiad fpc0 DFWE ERROR DFW: Filter : "loopback6-lo0.1-i" is NOT programmed in HW

The hardware showed this to be the case:

cmooney@lsw1-e1-eqiad> show pfe filter hw summary                                           

Slot 0

Unit:0:
Group                    Group-ID       Pipe      Allocated      Used      Free
-------------------------------------------------------------------------------
> Ingress filter groups:
  iRACL group            33             0         768            752       16
  iRACL group            34             1         768            752       16
  iRACL VxLAN group      239            0         768            11        757
  iRACL VxLAN group      240            1         768            11        757
> Egress filter groups:

{master:0}
cmooney@lsw1-e1-eqiad>

To keep the total TCAM usage within limits I therefore created a separate filter to be applied on non-default loopback interface units (i.e. those relating to VRFs), as most of the terms are not required in the overlay network.

Interestingly, as initially defined/applied, only allowing BGP/BFD for peers in the VRF/overlay network, the filter broke OSPF/BFD and BGP EVPN in the underlay network / global table. I am unsure how to explain this, but as a result I have included the "bgp-sessions" prefix-list in the allowed for BGP and BFD terms for the filter, and also allowed OSPF. With those additions to the filter applied to lo0.1 the OSPF/BGP/BFD sessions in the underlay came back up.

Total usage is now within limits but still fairly high:

cmooney@lsw1-e1-eqiad> show pfe filter hw summary    

Slot 0

Unit:0:
Group                    Group-ID       Pipe      Allocated      Used      Free
-------------------------------------------------------------------------------
> Ingress filter groups:
  iRACL group            33             0         768            540       228
  iRACL group            34             1         768            540       228
  iRACL VxLAN group      239            0         768            11        757
  iRACL VxLAN group      240            1         768            11        757

If we need to at some stage we can look at reversing this alignment of the filters to make them the same on CRs and switches, in which case we could have a switch-specific filter defined and leave out some of the terms.