Page MenuHomePhabricator

Add network policies to the ML k8s clusters
Closed, ResolvedPublic

Description

The current GlobalNetworkPolicies settings for ml-serve-{eqiad,codfw} clusters is empty, allowing any traffic to flow in/out the cluster without restrictions. This was good for initial testing, but now that we have reached a more stable phase we should add a base set of restrictions for ingress/egress and traffic flowing between pods.

Details

ProjectBranchLines +/-Subject
operations/deployment-chartsmaster+8 -5
operations/deployment-chartsmaster+18 -4
operations/deployment-chartsmaster+31 -1
operations/deployment-chartsmaster+14 -1
operations/deployment-chartsmaster+183 -132
operations/deployment-chartsmaster+52 -4
operations/deployment-chartsmaster+152 -3
operations/deployment-chartsmaster+8 -12
operations/deployment-chartsmaster+20 -20
operations/deployment-chartsmaster+2 -2
operations/deployment-chartsmaster+119 -4
operations/deployment-chartsmaster+6 -5
operations/deployment-chartsmaster+30 -8
operations/deployment-chartsmaster+35 -2
operations/deployment-chartsmaster+1 -0
operations/deployment-chartsmaster+25 -15
operations/deployment-chartsmaster+72 -4
operations/puppetproduction+10 -0
operations/deployment-chartsmaster+74 -115
Show related patches Customize query in gerrit

Event Timeline

It seems that the GlobalNetworkPolicies are split into two parts:

  • global ones (per cluster) that include things like allowing egress between each pod, allow DNS traffic to kube-system, etc..
  • per service ones, that include ingress filtering for each service (port exposed, etc..) and also egress to specific services if needed

I think that we should proceed with T286791 first, because we can currently only add per cluster rules.

List of ports used by various containers (got via nsenter):

  • istio ingress gateway:
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:8443            0.0.0.0:*               LISTEN      17095/envoy
tcp        0      0 0.0.0.0:15021           0.0.0.0:*               LISTEN      17095/envoy
tcp        0      0 0.0.0.0:8081            0.0.0.0:*               LISTEN      17095/envoy
tcp        0      0 0.0.0.0:15090           0.0.0.0:*               LISTEN      17095/envoy
tcp        0      0 127.0.0.1:15000         0.0.0.0:*               LISTEN      17095/envoy
tcp6       0      0 :::15020                :::*                    LISTEN      17046/pilot-agent
  • istio webhook (istiod)
tcp6       0      0 :::9090                 :::*                    LISTEN      61904/webhook       
tcp6       0      0 :::8008                 :::*                    LISTEN      61904/webhook       
tcp6       0      0 :::8443                 :::*                    LISTEN      61904/webhook
  • istio cluster local gateway
tcp        0      0 0.0.0.0:15090           0.0.0.0:*               LISTEN      6482/envoy          
tcp        0      0 127.0.0.1:15000         0.0.0.0:*               LISTEN      6482/envoy          
tcp        0      0 0.0.0.0:15021           0.0.0.0:*               LISTEN      6482/envoy          
tcp        0      0 0.0.0.0:8080            0.0.0.0:*               LISTEN      6482/envoy          
tcp6       0      0 :::15020                :::*                    LISTEN      6444/pilot-agent
  • knative activator
tcp6       0      0 :::8012                 :::*                    LISTEN      2436/activator      
tcp6       0      0 :::8013                 :::*                    LISTEN      2436/activator      
tcp6       0      0 :::9090                 :::*                    LISTEN      2436/activator      
tcp6       0      0 :::8008                 :::*                    LISTEN      2436/activator
  • knative autoscaler
tcp6       0      0 :::8080                 :::*                    LISTEN      2333/autoscaler     
tcp6       0      0 :::9090                 :::*                    LISTEN      2333/autoscaler     
tcp6       0      0 :::8008                 :::*                    LISTEN      2333/autoscaler
  • knative controller
tcp6       0      0 :::9090                 :::*                    LISTEN      2183/controller     
tcp6       0      0 :::8008                 :::*                    LISTEN      2183/controller
  • knative webhook
tcp6       0      0 :::8443                 :::*                    LISTEN      2048/webhook        
tcp6       0      0 :::9090                 :::*                    LISTEN      2048/webhook        
tcp6       0      0 :::8008                 :::*                    LISTEN      2048/webhook
  • istio networking
tcp6       0      0 :::9090                 :::*                    LISTEN      36626/controller    
tcp6       0      0 :::8008                 :::*                    LISTEN      36626/controller
  • kfserving-controller
tcp        0      0 127.0.0.1:8080          0.0.0.0:*               LISTEN      40307/manager       
tcp6       0      0 :::9443                 :::*                    LISTEN      40307/manager
  • revscoring
tcp        0      0 0.0.0.0:8080            0.0.0.0:*               LISTEN      41670/python3       
tcp6       0      0 :::9090                 :::*                    LISTEN      42058/queue         
tcp6       0      0 :::9091                 :::*                    LISTEN      42058/queue         
tcp6       0      0 :::8012                 :::*                    LISTEN      42058/queue         
tcp6       0      0 :::8080                 :::*                    LISTEN      41670/python3       
tcp6       0      0 :::8022                 :::*                    LISTEN      42058/queue

A lot of ports and complexity, but overall this should happen:

  • all pods shouldn't have rules for outgoing traffic (same as we do for the main cluster)
  • the istio gateway pods needs to be able to be contacted by all the pods to route traffic
  • the knative pods should be able to talk with each other, and they should accept traffic from kfserving's controller and istio (not from the ML service pods).
  • kfserving shouldn't really have any pods to be able to contact it

I had a chat with Janis this morning:

  • the GlobalNetworkPolicies that we define should be related to generic settings that are not tailored to a specific namespace etc.. basically only what it is common to all the pods.
  • we should add networking policies to the knative and kserve charts, and also probably to the kserve-inference chart as well.
  • the istio use case is a little different, since we don't really have a chart, more will come in collaboration with ServiceOps
  • when adding policies for a namespace, setting even one rule for egress or ingress will add deny-all-except kind of rules to the outbound or inbound traffic.

In theory we can add network policies in batches, and not all in once, that simplifies the complexity of the task.

Change 732677 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/deployment-charts@master] Add network policies for the kserve-inference chart deployments

https://gerrit.wikimedia.org/r/732677

Change 732677 merged by Elukey:

[operations/deployment-charts@master] Add network policies for the kserve-inference chart deployments

https://gerrit.wikimedia.org/r/732677

Change 732939 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/deployment-charts@master] kserve: add network policies

https://gerrit.wikimedia.org/r/732939

Change 735577 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] role::ml_k8s::master: add node-role.kubernetes.io/master labels

https://gerrit.wikimedia.org/r/735577

Change 735577 merged by Elukey:

[operations/puppet@production] role::ml_k8s::master: add node-role.kubernetes.io/master labels

https://gerrit.wikimedia.org/r/735577

Added some labels to the eqiad cluster:

  • node-role.kubernetes.io/master="" and node.kubernetes.io/disk-type=kvm to ml-serve-ctrl1001
  • node.kubernetes.io/disk-type=ssd to ml-serve1*

The labels were added manually via kubect label nodes ..., and also added to the kubelet's label list. The latter is a feature that allows admins to inject labels during cluster node registration time (so a one time thing). We cannot use the feature to add labels dynamically (we used kubect for it), but we can keep our list in sync if we have to bootstrap the cluster again. We still don't have a way in puppet to add labels dynamically (maybe via exec etc..), this combo should be enough for the moment.

The disk-type labels were added in T288345 by ServiceOps after ml-serve-eqiad was bootstrapped, so they were added to keep our clusters in sync and consistent with the kubelet's config (even if we don't really need the labels for the moment).

The master labels will be needed to implement some Calico policies like https://www.tigera.io/blog/securing-kubernetes-nodes-with-calico-automatic-host-endpoints/ or https://docs.projectcalico.org/security/kubernetes-nodes

Change 732939 merged by Elukey:

[operations/deployment-charts@master] kserve: add network policies

https://gerrit.wikimedia.org/r/732939

Change 737319 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/deployment-charts@master] kserve-inference: improve labels and network policy rules

https://gerrit.wikimedia.org/r/737319

Change 737320 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/deployment-charts@master] helmfile.d: add namespace to kserve's helmfile config

https://gerrit.wikimedia.org/r/737320

Change 737319 merged by Elukey:

[operations/deployment-charts@master] kserve-inference: improve labels and network policy rules

https://gerrit.wikimedia.org/r/737319

Change 737333 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/deployment-charts@master] helmfile.d: add basic egress GlobalNetworkPolicies for ml-serve

https://gerrit.wikimedia.org/r/737333

Change 737320 merged by Elukey:

[operations/deployment-charts@master] helmfile.d: add namespace to kserve's helmfile config

https://gerrit.wikimedia.org/r/737320

Change 737333 merged by Elukey:

[operations/deployment-charts@master] helmfile.d: add basic egress GlobalNetworkPolicies for ml-serve

https://gerrit.wikimedia.org/r/737333

Change 737399 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/deployment-charts@master] kserve: add missing egress policies for the controller

https://gerrit.wikimedia.org/r/737399

Change 737400 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/deployment-charts@master] knative-serving: add basic egress network policies

https://gerrit.wikimedia.org/r/737400

Change 737399 merged by Elukey:

[operations/deployment-charts@master] kserve: add missing egress policies for the controller

https://gerrit.wikimedia.org/r/737399

Change 737414 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/deployment-charts@master] kserve: move labels from StatefulSet to its pod template

https://gerrit.wikimedia.org/r/737414

Change 737414 merged by Elukey:

[operations/deployment-charts@master] kserve: move labels from StatefulSet to its pod template

https://gerrit.wikimedia.org/r/737414

Change 737400 merged by Elukey:

[operations/deployment-charts@master] knative-serving: add basic egress network policies

https://gerrit.wikimedia.org/r/737400

Change 737421 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/deployment-charts@master] knative-serving: add a name to the controller's network policy

https://gerrit.wikimedia.org/r/737421

Change 737421 merged by Elukey:

[operations/deployment-charts@master] knative-serving: add a name to the controller's network policy

https://gerrit.wikimedia.org/r/737421

Change 737428 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/deployment-charts@master] knative-serving: move labels to pod templates

https://gerrit.wikimedia.org/r/737428

Change 737428 merged by Elukey:

[operations/deployment-charts@master] knative-serving: move labels to pod templates

https://gerrit.wikimedia.org/r/737428

Change 737432 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/deployment-charts@master] helmfile.d: move Docker registry's IPs to ml-serve.yaml

https://gerrit.wikimedia.org/r/737432

Change 737432 merged by Elukey:

[operations/deployment-charts@master] helmfile.d: move Docker registry's IPs to ml-serve.yaml

https://gerrit.wikimedia.org/r/737432

Some updates:

  • fixed label targeting for the kserve-inference chart, and added a specific rule for the queue-proxy container.
  • added basic GlobalNetworkPolicies to calico's config
  • added ingress/egress policies for kserve
  • added egress policies for knative-serving (k8s api, docker-registry).

Next inline:

  • complete network policies for knative-serving
  • add network policies for istio
  • complete GlobalNetworkPolicies (add default-deny)

Change 737875 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/deployment-charts@master] knative-serving: add ingress network policies

https://gerrit.wikimedia.org/r/737875

Change 737875 merged by Elukey:

[operations/deployment-charts@master] knative-serving: add ingress network policies

https://gerrit.wikimedia.org/r/737875

Change 738355 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/deployment-charts@master] kserve-inference: improve network policies

https://gerrit.wikimedia.org/r/738355

Change 738355 merged by Elukey:

[operations/deployment-charts@master] kserve-inference: improve network policies

https://gerrit.wikimedia.org/r/738355

Change 738360 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/deployment-charts@master] knative-serving: improve network policies

https://gerrit.wikimedia.org/r/738360

Change 738360 merged by jenkins-bot:

[operations/deployment-charts@master] knative-serving: improve network policies

https://gerrit.wikimedia.org/r/738360

Change 738438 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/deployment-charts@master] knative-serving: allow networking-istio to contact the k8s api

https://gerrit.wikimedia.org/r/738438

Change 738438 merged by jenkins-bot:

[operations/deployment-charts@master] knative-serving: allow networking-istio to contact the k8s api

https://gerrit.wikimedia.org/r/738438

Change 738446 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/deployment-charts@master] knative-serving: allow webhook pods to contact the k8s api

https://gerrit.wikimedia.org/r/738446

Change 738446 merged by Elukey:

[operations/deployment-charts@master] knative-serving: allow webhook pods to contact the k8s api

https://gerrit.wikimedia.org/r/738446

Change 739791 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/deployment-charts@master] helmfile.d: add default-deny and icmp to ml-serve's settings

https://gerrit.wikimedia.org/r/739791

Change 739791 merged by Elukey:

[operations/deployment-charts@master] helmfile.d: add default-deny and icmp to ml-serve's settings

https://gerrit.wikimedia.org/r/739791

Istio policies applied, plus global default-deny (same used in other clusters) applied. Deleted all the pods, they came back up correctly!

Change 740804 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/deployment-charts@master] knative-serving: add stricter network policies for the activator pod

https://gerrit.wikimedia.org/r/740804

Change 740804 merged by Elukey:

[operations/deployment-charts@master] knative-serving: add stricter network policies for the activator pod

https://gerrit.wikimedia.org/r/740804

elukey claimed this task.