Page MenuHomePhabricator

CloudVPS: introduce filtering for neutron BGP addresses
Closed, ResolvedPublic

Description

This task is to track the work to introduce filtering for neutron BGP addresses.

Reported by diffscan: eno2-2120.cloudnet2002-dev.wikimedia.org and eno2-2120.cloudnet2003-dev.wikimedia.org, ports 22 and 9100

NOTE: the FQDN is now br-external.cloudnet2002-dev.wikimedia.org and br-external.cloudnet2003-dev.wikimedia.org but the situation is the same.

Event Timeline

aborrero created this task.Mar 4 2020, 11:44 AM
aborrero added a comment.EditedMar 4 2020, 1:07 PM

This is my initial proposal:

diff -u router.org router.new  --color
--- router.org	2020-03-04 12:45:50.139827978 +0100
+++ router.new	2020-03-04 14:06:26.158971812 +0100
@@ -804,6 +804,7 @@
             family inet {
                 filter {
                     input cloud-in4;
+                    output cloud-out4;
                 }
                 address 208.80.153.187/29 {
                     vrrp-group 121 {
@@ -3199,6 +3200,34 @@
                 then accept;
             }
         }
+        /* Applied to cloud instance traffic entering the cloud-instance-transport vlan */
+        filter cloud-out4 {
+            interface-specific;
+            term cloudnet-accept-bgp {
+                from {
+                    destination-address {
+                        /* br-external.cloudnet2002-dev.wikimedia.org */
+                        208.80.153.188/32;
+                        /* br-external.cloudnet2003-dev.wikimedia.org */
+                        208.80.153.189/32;
+                    }
+                    protocol tcp;
+                    destination-port bgp;
+                }
+                then accept;
+            }        
+            term cloudnet-else-drop {
+                from {
+                    destination-address {
+                        /* br-external.cloudnet2002-dev.wikimedia.org */
+                        208.80.153.188/32;
+                        /* br-external.cloudnet2003-dev.wikimedia.org */
+                        208.80.153.189/32;
+                    }
+                }
+                then discard;
+            }        
+        }
         /* Applied to cloud instance traffic going out of the cloud-instance-transport vlan. Last audit in T199435 and T211921. */
         filter cloud-in4 {
             interface-specific;

please @ayounsi review and double-check.

aborrero claimed this task.Mar 4 2020, 1:09 PM
aborrero moved this task from Inbox to Doing on the cloud-services-team (Kanban) board.
aborrero updated the task description. (Show Details)Mar 5 2020, 1:09 PM

Ping. It would be nice to address this ASAP.

ayounsi added a comment.EditedMar 6 2020, 2:21 PM

Will push the following to keep the previous behavior and be on a whitelist basis instead of blacklist. We can tune it later on if needed.

[edit firewall family inet]
      filter cloud-in4 { ... }
+     /* T246887 */
+     filter cloud-out4 {
+         interface-specific;
+         term allow-local {
+             from {
+                 source-address {
+                     208.80.153.184/29;
+                 }
+             }
+             then accept;
+         }
+         term allow-cloudnet-gw {
+             from {
+                 destination-address {
+                     208.80.153.190/32;
+                 }
+             }
+             then accept;
+         }
+         term deny-local {
+             from {
+                 destination-address {
+                     208.80.153.184/29;
+                 }
+             }
+             then discard;
+         }
+     }

Mentioned in SAL (#wikimedia-operations) [2020-03-06T14:44:56Z] <XioNoX> add cloud-out4 firewall filter in codfw - T246887

Confirmed that nc -zv 208.80.153.189 22 doesn't work anymore. While ping to 208.80.153.190 does.

I'll send a Homer CR to make it generic across both sites.

Change 577575 had a related patch set uploaded (by Ayounsi; owner: Ayounsi):
[operations/homer/public@master] Add cloud-out4 firewall filter

https://gerrit.wikimedia.org/r/577575

aborrero triaged this task as High priority.Mar 11 2020, 9:42 AM

This filter might be blocking legit traffic.

I believe we need to explicitly allow traffic with origin/destination our neutron addresses:

  • routing_source_ip: 185.15.57.1 in codfw1dev
  • routing_source_ip: 185.15.56.1 in eqiad1

Those are the addresses all outgoing traffic uses as source NAT, except traffic using floating IPs which also may be an overlapping setting anyway:

  • floating IPs in codfw1dev: 185.15.57.0/29
  • floating IPs in eqiad1: 185.15.56.0/25

I think I may have a solution for this, stay tuned.

Mentioned in SAL (#wikimedia-cloud) [2020-03-11T12:50:56Z] <arturo> [codfw1dev] several tests creating/deleting address scopes (T244727 T247135 T246887 T245606)

Change 577575 abandoned by Ayounsi:
Add cloud-out4 firewall filter

Reason:
Experiment not successful

https://gerrit.wikimedia.org/r/577575

aborrero closed this task as Resolved.Mar 18 2020, 11:09 AM

We decided to drop the BGP setup for now.