Page MenuHomePhabricator

Investigate unicast RPF loose mode
Closed, ResolvedPublic

Description

During the previous DDoS we saw inbound traffic from AS0 in Turnilo, which mean Pmacct didn't have a matching routing table entry to associate the source IP with.
See for example: https://w.wiki/GaH
This can be either some lag between Pmacct and the routing table (unlikely) or traffic from spoofed IPs, using ranges not present in the DFZ.

There is a tool to drop this kind of traffic at our borders: uRPF loose mode.

The concept is easy, for selected interfaces (public facing) the router will check if the source IP is covered by any prefix in its routing table. If not, it discards it.

There are a few knobs to tweak to make it less scary:

  • loose mode: by default, strict mode will check if the packet is coming from the interface that is also the outbound route to this source IP (symmetric routing), which we don't want here. Thus the loose mode, that looks if there is a route along all the ones in the routing table.
  • fail-filter: allow us to apply a firewall rule when a packet is flagged by uRPF, in our case we would first log them to see if there would be any false positive
  • unicast-reverse-path feasible-paths: the default is to only look at active paths, which could potentially drop traffic during routing re-convergence.

You can find more details on https://www.juniper.net/documentation/en_US/junos/topics/task/configuration/interfaces-configuring-unicast-rpf.html

There is at least one report of it not working as well as it's supposed to.

If it does work it would be a great tool to reduce the impact of both amplification and flood.

Here is the first step:

[edit groups external-links interfaces <*> unit <*> family inet]
+        rpf-check {
+            fail-filter log-only4;
+            mode loose;
+        }
[edit groups external-links interfaces <*> unit <*> family inet6]
+        rpf-check {
+            fail-filter log-only6;
+            mode loose;
+        }
[edit forwarding-options]
+   rpf-loose-mode-discard {
+       family {
+           inet;
+           inet6;
+       }
+   }
[edit routing-options forwarding-table]
+   unicast-reverse-path feasible-paths;
[edit firewall family inet]
      filter vrrp-in4 { ... }
+     filter log-only4 {
+         term default {
+             then {
+                 log;
+                 accept;
+             }
+         }
+     }
[edit firewall family inet6]
      filter vrrp-in6 { ... }
+     filter log-only6 {
+         term default {
+             then {
+                 log;
+                 accept;
+             }
+         }
+     }

Then make a Kibana dashboard
Then if all good, change the fail filter action to sample and discard

Event Timeline

ayounsi created this task.
ayounsi created this object with visibility "acl*sre-team (Project)".

If for some reasons it doesn't work out, then we should have some alerting on spike of inbound traffic with source_as=0.

Do we already use something like the team cymru bogon feed, it may catch some of the low hanging fruit https://www.team-cymru.com/bogon-reference-bgp.html

The main issue I see here is the dependency on Cymru, if they start sending us valid prefixes for some reason, then we will blackhole legit traffic.
It's also external, so if for any reason the BGP session to them dies, then we won't have that layer of security.
Last it's less dynamic, they only are about bogons, but not prefixes assigned but unused.

I think the proposal as it stands now sounds good in paticuler i fully agree with feasible-paths and loose mode and it makes a lot of senses to enable this with logging enabled before we start dropping

I think the proposal as it stands now sounds good in paticuler i fully agree with feasible-paths and loose mode and it makes a lot of senses to enable this with logging enabled before we start dropping

+1 to all of this

For the record, this got pushed to cr3-ulsfo for v6 external links then v4.

Taking one transit interface:

ayounsi@cr3-ulsfo# run show interfaces xe-0/1/0 extensive | match RPF
      Flags: Sendbcast-pkt-to-re, Is-Primary, uRPF, uRPF-loose
      RPF Failures: Packets: 519, Bytes: 35764
      Flags: Is-Primary, uRPF, uRPF-loose
      RPF Failures: Packets: 118, Bytes: 17500

The first two lines are for v4, the last 2 are for v6. So packets are being caught by the filter.

The log action is different from the syslog one, as it writes to a temporary buffer.
As it's not too verbose, we can change it to syslog to for example dashboard it.
Or eventually to "count" if we want to graph it using a SNMP counters to Grafana relay.


Quick parenthesis:
Running the following:
ayounsi@cr3-ulsfo# run show firewall log
Interestingly show packets being dropped. See bellow:

09:05:57  pfe       D      xe-0/1/0.0    TCP             10.6.4.98                        198.35.26.96
09:05:56  pfe       D      xe-0/1/0.0    ICMP            10.255.255.3                     198.35.26.244
09:05:47  pfe       D      xe-0/1/0.0    TCP             172.16.72.125                    198.35.26.112
09:10:49  pfe       D      xe-0/1/0.0    TCP             192.168.0.81                     198.35.26.112

So those for example are all private IPs, that shouldn't be routed on the Internet.

08:56:59  pfe       D      xe-0/1/0.0    TCP             100.126.x.y                   198.35.26.96
09:13:06  pfe       D      xe-0/1/0.0    ICMP            100.99.x.y                       198.35.26.244

Bogons.

08:57:03  pfe       D      xe-0/1/0.0    ICMPv6          ::ffff:185.96.x.y             2620:0:863:201:198:35:26:244
09:10:46  pfe       D      xe-0/1/0.0    ICMPv6          fcaa::17                         2620:0:863:201:198:35:26:244

IPv6 bogons

At first I thought those were dropped by the uRPF filter, but is has a permit policy.
They are actually packets that match the discard action of the border-in4/6 filters.
So I don't know if it's a bug that cause them to show up because they match both filters. But it's not an issue.


Focusing on the uRPF filter, and showing logs for ayounsi@cr3-ulsfo> show firewall log | match "A "

10:01:25  pfe       A      xe-0/1/0.0    ICMP            37.49.x.y                     198.35.26.244

Packets from RPKI invalid prefixes.

10:01:31  pfe       A      xe-0/1/2.0    TCP             194.68.x.y                     198.35.26.227
10:01:30  pfe       A      xe-0/1/2.0    ICMP            119.27.x.y                    198.35.26.244
10:01:27  pfe       A      xe-0/1/2.0    ICMPv6          2001:506:x:y::1              2620:0:863:ed1a::1
10:01:33  pfe       A      xe-0/1/2.0    ICMP            80.81.x.y                     198.35.26.244
10:02:02  pfe       A      xe-0/1/0.0    ICMP            195.66.x.y                   198.35.26.244

Prefix not present in the DFZ (from RIPE looking glass).

This quick manual pass shows doesn't show any false positive.

Added the config knob to Homer with https://gerrit.wikimedia.org/r/c/587281
Deployed uRPF log only to cr4-ulsfo as well.

Next steps are:

  1. Parse firewall logs in ELK with https://gerrit.wikimedia.org/r/c/587265
  2. Create temporary Kibana dashboard
  3. Progressively deploy uRPF log to syslog infra wide for 24h or so (as firewall logs are quite spammy)
  4. Use dashboard to dive in logs, see if any false positive show up
  5. If no red flag, progressively update uRPF fleet wide to sample then discard hits
  6. Discard dashboard
  7. Update Homer to remove the feature rollout flag

So I don't know if it's a bug that cause them to show up because they match both filters. But it's not an issue.

Note that this doesn't happen on the MX480s.

All routers have been logging hits to syslog since 11am UTC.

So far a few observations:

  • There is one curious case of traffic from Cloudflare's IPs not present in the DFZ (CF's NOC emailed)

Ignoring CF:

  • ICMPv6 is surprisingly at equal levels with TCP

Ignoring ICMP (v4 and v6):

  • It's stable at around 10000/h (167/h, 2.7/s)
  • Most hits are from RPKI invalids (and they seem to be changing, aka people break and fix their miss-configured ROAs)
  • The reminders of hits are from prefixes that are either
    • Fully gone from the DFZ
    • Only have a few RIS peers seeing it (which can be explained in a lot of different reasons, mostly miss-configurations)

It would be interesting to drill more through the data.

For the scope of this task, I couldn't find any false positive. In addition, the rate of errors seems to match with what I'd expect from both providers miss-configurations (RPKI, routing) and overall Internet background noise (spam, tests) towards Wikipedia.

Turning off log to syslog to not flood syslog too much as we have all the data we need.

Default changed to sample + discard on all routers.

ayounsi changed the visibility from "acl*sre-team (Project)" to "Public (No Login Required)".Apr 20 2020, 10:31 AM