The current GlobalNetworkPolicies settings for ml-serve-{eqiad,codfw} clusters is empty, allowing any traffic to flow in/out the cluster without restrictions. This was good for initial testing, but now that we have reached a more stable phase we should add a base set of restrictions for ingress/egress and traffic flowing between pods.
Description
Description
Details
Details
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | None | T272917 Lift Wing proof of concept | |||
| Resolved | elukey | T289834 Add network policies to the ML k8s clusters |
Event Timeline
Comment Actions
It seems that the GlobalNetworkPolicies are split into two parts:
- global ones (per cluster) that include things like allowing egress between each pod, allow DNS traffic to kube-system, etc..
- per service ones, that include ingress filtering for each service (port exposed, etc..) and also egress to specific services if needed
I think that we should proceed with T286791 first, because we can currently only add per cluster rules.
Comment Actions
List of ports used by various containers (got via nsenter):
- istio ingress gateway:
Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 0.0.0.0:8443 0.0.0.0:* LISTEN 17095/envoy tcp 0 0 0.0.0.0:15021 0.0.0.0:* LISTEN 17095/envoy tcp 0 0 0.0.0.0:8081 0.0.0.0:* LISTEN 17095/envoy tcp 0 0 0.0.0.0:15090 0.0.0.0:* LISTEN 17095/envoy tcp 0 0 127.0.0.1:15000 0.0.0.0:* LISTEN 17095/envoy tcp6 0 0 :::15020 :::* LISTEN 17046/pilot-agent
- istio webhook (istiod)
tcp6 0 0 :::9090 :::* LISTEN 61904/webhook tcp6 0 0 :::8008 :::* LISTEN 61904/webhook tcp6 0 0 :::8443 :::* LISTEN 61904/webhook
- istio cluster local gateway
tcp 0 0 0.0.0.0:15090 0.0.0.0:* LISTEN 6482/envoy tcp 0 0 127.0.0.1:15000 0.0.0.0:* LISTEN 6482/envoy tcp 0 0 0.0.0.0:15021 0.0.0.0:* LISTEN 6482/envoy tcp 0 0 0.0.0.0:8080 0.0.0.0:* LISTEN 6482/envoy tcp6 0 0 :::15020 :::* LISTEN 6444/pilot-agent
- knative activator
tcp6 0 0 :::8012 :::* LISTEN 2436/activator tcp6 0 0 :::8013 :::* LISTEN 2436/activator tcp6 0 0 :::9090 :::* LISTEN 2436/activator tcp6 0 0 :::8008 :::* LISTEN 2436/activator
- knative autoscaler
tcp6 0 0 :::8080 :::* LISTEN 2333/autoscaler tcp6 0 0 :::9090 :::* LISTEN 2333/autoscaler tcp6 0 0 :::8008 :::* LISTEN 2333/autoscaler
- knative controller
tcp6 0 0 :::9090 :::* LISTEN 2183/controller tcp6 0 0 :::8008 :::* LISTEN 2183/controller
- knative webhook
tcp6 0 0 :::8443 :::* LISTEN 2048/webhook tcp6 0 0 :::9090 :::* LISTEN 2048/webhook tcp6 0 0 :::8008 :::* LISTEN 2048/webhook
- istio networking
tcp6 0 0 :::9090 :::* LISTEN 36626/controller tcp6 0 0 :::8008 :::* LISTEN 36626/controller
- kfserving-controller
tcp 0 0 127.0.0.1:8080 0.0.0.0:* LISTEN 40307/manager tcp6 0 0 :::9443 :::* LISTEN 40307/manager
- revscoring
tcp 0 0 0.0.0.0:8080 0.0.0.0:* LISTEN 41670/python3 tcp6 0 0 :::9090 :::* LISTEN 42058/queue tcp6 0 0 :::9091 :::* LISTEN 42058/queue tcp6 0 0 :::8012 :::* LISTEN 42058/queue tcp6 0 0 :::8080 :::* LISTEN 41670/python3 tcp6 0 0 :::8022 :::* LISTEN 42058/queue
Comment Actions
A lot of ports and complexity, but overall this should happen:
- all pods shouldn't have rules for outgoing traffic (same as we do for the main cluster)
- the istio gateway pods needs to be able to be contacted by all the pods to route traffic
- the knative pods should be able to talk with each other, and they should accept traffic from kfserving's controller and istio (not from the ML service pods).
- kfserving shouldn't really have any pods to be able to contact it
Comment Actions
I had a chat with Janis this morning:
- the GlobalNetworkPolicies that we define should be related to generic settings that are not tailored to a specific namespace etc.. basically only what it is common to all the pods.
- we should add networking policies to the knative and kserve charts, and also probably to the kserve-inference chart as well.
- the istio use case is a little different, since we don't really have a chart, more will come in collaboration with ServiceOps
- when adding policies for a namespace, setting even one rule for egress or ingress will add deny-all-except kind of rules to the outbound or inbound traffic.
In theory we can add network policies in batches, and not all in once, that simplifies the complexity of the task.
Comment Actions
Change 732677 had a related patch set uploaded (by Elukey; author: Elukey):
[operations/deployment-charts@master] Add network policies for the kserve-inference chart deployments
Comment Actions
Change 732677 merged by Elukey:
[operations/deployment-charts@master] Add network policies for the kserve-inference chart deployments
Comment Actions
Change 732939 had a related patch set uploaded (by Elukey; author: Elukey):
[operations/deployment-charts@master] kserve: add network policies
Comment Actions
Change 735577 had a related patch set uploaded (by Elukey; author: Elukey):
[operations/puppet@production] role::ml_k8s::master: add node-role.kubernetes.io/master labels
Comment Actions
Change 735577 merged by Elukey:
[operations/puppet@production] role::ml_k8s::master: add node-role.kubernetes.io/master labels
Comment Actions
Added some labels to the eqiad cluster:
- node-role.kubernetes.io/master="" and node.kubernetes.io/disk-type=kvm to ml-serve-ctrl1001
- node.kubernetes.io/disk-type=ssd to ml-serve1*
The labels were added manually via kubect label nodes ..., and also added to the kubelet's label list. The latter is a feature that allows admins to inject labels during cluster node registration time (so a one time thing). We cannot use the feature to add labels dynamically (we used kubect for it), but we can keep our list in sync if we have to bootstrap the cluster again. We still don't have a way in puppet to add labels dynamically (maybe via exec etc..), this combo should be enough for the moment.
The disk-type labels were added in T288345 by ServiceOps after ml-serve-eqiad was bootstrapped, so they were added to keep our clusters in sync and consistent with the kubelet's config (even if we don't really need the labels for the moment).
The master labels will be needed to implement some Calico policies like https://www.tigera.io/blog/securing-kubernetes-nodes-with-calico-automatic-host-endpoints/ or https://docs.projectcalico.org/security/kubernetes-nodes
Comment Actions
Change 732939 merged by Elukey:
[operations/deployment-charts@master] kserve: add network policies
Comment Actions
Change 737319 had a related patch set uploaded (by Elukey; author: Elukey):
[operations/deployment-charts@master] kserve-inference: improve labels and network policy rules
Comment Actions
Change 737320 had a related patch set uploaded (by Elukey; author: Elukey):
[operations/deployment-charts@master] helmfile.d: add namespace to kserve's helmfile config
Comment Actions
Change 737319 merged by Elukey:
[operations/deployment-charts@master] kserve-inference: improve labels and network policy rules
Comment Actions
Change 737333 had a related patch set uploaded (by Elukey; author: Elukey):
[operations/deployment-charts@master] helmfile.d: add basic egress GlobalNetworkPolicies for ml-serve
Comment Actions
Change 737320 merged by Elukey:
[operations/deployment-charts@master] helmfile.d: add namespace to kserve's helmfile config
Comment Actions
Change 737333 merged by Elukey:
[operations/deployment-charts@master] helmfile.d: add basic egress GlobalNetworkPolicies for ml-serve
Comment Actions
Change 737399 had a related patch set uploaded (by Elukey; author: Elukey):
[operations/deployment-charts@master] kserve: add missing egress policies for the controller
Comment Actions
Change 737400 had a related patch set uploaded (by Elukey; author: Elukey):
[operations/deployment-charts@master] knative-serving: add basic egress network policies
Comment Actions
Change 737399 merged by Elukey:
[operations/deployment-charts@master] kserve: add missing egress policies for the controller
Comment Actions
Change 737414 had a related patch set uploaded (by Elukey; author: Elukey):
[operations/deployment-charts@master] kserve: move labels from StatefulSet to its pod template
Comment Actions
Change 737414 merged by Elukey:
[operations/deployment-charts@master] kserve: move labels from StatefulSet to its pod template
Comment Actions
Change 737400 merged by Elukey:
[operations/deployment-charts@master] knative-serving: add basic egress network policies
Comment Actions
Change 737421 had a related patch set uploaded (by Elukey; author: Elukey):
[operations/deployment-charts@master] knative-serving: add a name to the controller's network policy
Comment Actions
Change 737421 merged by Elukey:
[operations/deployment-charts@master] knative-serving: add a name to the controller's network policy
Comment Actions
Change 737428 had a related patch set uploaded (by Elukey; author: Elukey):
[operations/deployment-charts@master] knative-serving: move labels to pod templates
Comment Actions
Change 737428 merged by Elukey:
[operations/deployment-charts@master] knative-serving: move labels to pod templates
Comment Actions
Change 737432 had a related patch set uploaded (by Elukey; author: Elukey):
[operations/deployment-charts@master] helmfile.d: move Docker registry's IPs to ml-serve.yaml
Comment Actions
Change 737432 merged by Elukey:
[operations/deployment-charts@master] helmfile.d: move Docker registry's IPs to ml-serve.yaml
Comment Actions
Some updates:
- fixed label targeting for the kserve-inference chart, and added a specific rule for the queue-proxy container.
- added basic GlobalNetworkPolicies to calico's config
- added ingress/egress policies for kserve
- added egress policies for knative-serving (k8s api, docker-registry).
Next inline:
- complete network policies for knative-serving
- add network policies for istio
- complete GlobalNetworkPolicies (add default-deny)
Comment Actions
Change 737875 had a related patch set uploaded (by Elukey; author: Elukey):
[operations/deployment-charts@master] knative-serving: add ingress network policies
Comment Actions
Change 737875 merged by Elukey:
[operations/deployment-charts@master] knative-serving: add ingress network policies
Comment Actions
Change 738355 had a related patch set uploaded (by Elukey; author: Elukey):
[operations/deployment-charts@master] kserve-inference: improve network policies
Comment Actions
Change 738355 merged by Elukey:
[operations/deployment-charts@master] kserve-inference: improve network policies
Comment Actions
Change 738360 had a related patch set uploaded (by Elukey; author: Elukey):
[operations/deployment-charts@master] knative-serving: improve network policies
Comment Actions
Change 738360 merged by jenkins-bot:
[operations/deployment-charts@master] knative-serving: improve network policies
Comment Actions
Change 738438 had a related patch set uploaded (by Elukey; author: Elukey):
[operations/deployment-charts@master] knative-serving: allow networking-istio to contact the k8s api
Comment Actions
Change 738438 merged by jenkins-bot:
[operations/deployment-charts@master] knative-serving: allow networking-istio to contact the k8s api
Comment Actions
Change 738446 had a related patch set uploaded (by Elukey; author: Elukey):
[operations/deployment-charts@master] knative-serving: allow webhook pods to contact the k8s api
Comment Actions
Change 738446 merged by Elukey:
[operations/deployment-charts@master] knative-serving: allow webhook pods to contact the k8s api
Comment Actions
Change 739791 had a related patch set uploaded (by Elukey; author: Elukey):
[operations/deployment-charts@master] helmfile.d: add default-deny and icmp to ml-serve's settings
Comment Actions
Change 739791 merged by Elukey:
[operations/deployment-charts@master] helmfile.d: add default-deny and icmp to ml-serve's settings
Comment Actions
Istio policies applied, plus global default-deny (same used in other clusters) applied. Deleted all the pods, they came back up correctly!
Comment Actions
Change 740804 had a related patch set uploaded (by Elukey; author: Elukey):
[operations/deployment-charts@master] knative-serving: add stricter network policies for the activator pod
Comment Actions
Change 740804 merged by Elukey:
[operations/deployment-charts@master] knative-serving: add stricter network policies for the activator pod