Page MenuHomePhabricator

Review filtering for cloud-hosts on CR routers eqiad
Closed, ResolvedPublic

Description

Vlan 1118 (cloud-hosts1-eqiad) in eqiad is used to connect physical servers deployed for WMCS to production networks. Specifically this network is for physical hosts providing the virtualization layer, connected on a separate physical interface than the internal cloud networks / those for VM guests. It's the "management network" described in case 4 here:

https://wikitech.wikimedia.org/wiki/Cross-Realm_traffic_guidelines#Case_4:_using_isolation_mechanisms

Currently the gateway for this Vlan is the CR routers in eqiad. These have an ACL/filter applied to restrict traffic coming from cloud hosts ("labs-v4" and "labs-v6"). During the recent work to move filters to Capirca it became clear a thorough review of these lists was warranted.

Is filtering required?

The cloud hosts themselves are responsible for isolation of guest VM and internal networks, and thus only trusted traffic from properly secured hosts under WMF control should arrive on this vlan. It is not unreasonable to conclude that no filter is required on these interfaces for that reason.

That said the CR routers do represent a control point, and security-in-depth is good practice, so it probably makes sense to deploy some filtering at that point. A balance between sensible security, convenience for the cloud team and level of maintenance for netops should probably be struck.

uRPF

WMF has used strict uRPF filtering on certain interfaces before (see T266561). It may make sense to enable it on the sub-interfaces for this Vlan also, which might also simplify the ruleset needed in the filter.

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald Transcript
jbond triaged this task as Medium priority.Jun 24 2021, 1:20 PM
jbond added a project: cloud-services-team.
jbond added a subscriber: aborrero.

+1 to me. As the end goal is *less* filtering for WMCS hosts, it's a win-win.

To me the next steps are:

Change 702446 had a related patch set uploaded (by Cathal Mooney; author: Cathal Mooney):

[operations/homer/public@master] Added optional ability to enable uRPF filtering on arbitary CR ints

https://gerrit.wikimedia.org/r/702446

Change 701347 had a related patch set uploaded (by Ayounsi; author: Ayounsi):

[operations/homer/public@master] Port labs-in4/6 to Capirca

https://gerrit.wikimedia.org/r/701347

Change 701347 merged by jenkins-bot:

[operations/homer/public@master] Port labs-in4/6 to Capirca

https://gerrit.wikimedia.org/r/701347

Change 774478 had a related patch set uploaded (by Ayounsi; author: Ayounsi):

[operations/homer/public@master] Apply strict uRPF to the cloud-hosts vlan

https://gerrit.wikimedia.org/r/774478

I pushed the following temporarily and confirmed that no traffic is hitting the filter:

cr1-eqiad (vrrp gw)
[edit interfaces xe-3/0/4 unit 1118 family inet]
+       rpf-check fail-filter log-only4;
[edit firewall family inet]
+     filter log-only4 {
+         term default {
+             then {
+                 syslog;
+                 accept;
+             }
+         }
+     }
      filter transport-in4 { ... }

Change 702446 merged by jenkins-bot:

[operations/homer/public@master] Added optional ability to enable uRPF filtering on arbitary CR ints

https://gerrit.wikimedia.org/r/702446

Change 774478 merged by jenkins-bot:

[operations/homer/public@master] Apply strict uRPF to the cloud-hosts vlan

https://gerrit.wikimedia.org/r/774478

Mentioned in SAL (#wikimedia-operations) [2022-03-30T11:19:09Z] <XioNoX> apply urpf strict filter to eqiad cloud-hosts vlan - T285461

Mentioned in SAL (#wikimedia-cloud) [2022-03-30T11:20:07Z] <arturo> apply urpf strict filter to eqiad cloud-hosts vlan - T285461

Change 775279 had a related patch set uploaded (by Ayounsi; author: Ayounsi):

[operations/homer/public@master] Enable urpf strict on codfw cloud-hosts

https://gerrit.wikimedia.org/r/775279

Change 775279 merged by jenkins-bot:

[operations/homer/public@master] Enable urpf strict on codfw cloud-hosts

https://gerrit.wikimedia.org/r/775279

ayounsi claimed this task.

All done here!

Change 776973 had a related patch set uploaded (by Ayounsi; author: Ayounsi):

[operations/homer/public@master] uRPF: add DHCP exception

https://gerrit.wikimedia.org/r/776973

Change 776973 merged by jenkins-bot:

[operations/homer/public@master] uRPF: add DHCP exception

https://gerrit.wikimedia.org/r/776973

Mentioned in SAL (#wikimedia-operations) [2022-04-04T17:25:41Z] <XioNoX> push urpf DHCP exception to all core routers with urpf configured - T285461

Unfortunately the uRPF exception command is not supported on the QFX platform, which means configuring it on top-of-rack irb/vlan interfaces facing end hosts is not an option, as they need DHCP. Just wanted to mention here for informational purposes.

cmooney@lsw1-e2-eqiad> show configuration interfaces irb unit 1040 family inet   
/* T285461 */
##
## Warning: statement ignored: unsupported platform (qfx5120-48y-8c)
##
rpf-check fail-filter rpf-exceptions4;
address 10.64.139.1/24;
{master:0}[edit interfaces irb unit 1040 family inet]
cmooney@lsw1-e2-eqiad# set rpf-check ?
Possible completions:
  <[Enter]>            Execute this command
+ apply-groups         Groups from which to inherit configuration data
+ apply-groups-except  Don't inherit configuration data from these groups
> mode                 Mode for reverse path forwarding
  |                    Pipe through a command
{master:0}[edit interfaces irb unit 1040 family inet]
cmooney@lsw1-e2-eqiad#