Page MenuHomePhabricator
Create Task
Maniphest T287491

Allow to address Kubernetes API servers from NetworkPolicy
Open, HighPublic
Actions

Description

Currently it's not possible to address Kubernets API servers easily from NetworkPolicy, but starting with T287443 we now need to.

Right now we have the list of IPs hardcoded in various

The reason behind this is that Calico does not model the Endpoints of the k8s service kubernetes.default.svc.cluster.local as WorkloadEndpoints as they are not backed by Pods. In version 3.20 and onwards concept of service-based egress rules was introduced, especially mentioning the ability to define rules for services not backed by pods.

The following chart/services will need to be refactored/adapted:

  • helmfile_istio-gateways.yaml
  • flink-kubernetes-operator
  • spark-operator
  • cert-manager
  • kserve
  • knative-serving

Details

SubjectRepoBranchLines +/-
Remove kubernetesMasters definition from all wikikube valuesoperations/deployment-chartsmaster+0 -21
Remove kubernetesMasters definition from dse-k8soperations/deployment-chartsmaster+1 -7
rdf-streaming-updater: Remove duplicate definition of k8s and zkoperations/deployment-chartsmaster+1 -35
Remove kubernetesMasters definition from staging-codfwoperations/deployment-chartsmaster+0 -7
cirrus-streaming-updater: remove zk network policiesoperations/deployment-chartsmaster+0 -11
flink-kubernetes-operator: Remove dependency on kubernetesMasters.cidrsoperations/deployment-chartsmaster+53 -50
admin_ng/helmfile_istio-gateway: Remove dependency on kubernetesMasters.cidrsoperations/deployment-chartsmaster+40 -41
admin_ng/cert-manager: Remove dependency on kubernetesMasters.cidrsoperations/deployment-chartsmaster+28 -29
admin_ng/cert-manager: Remove dependency on kubernetesMasters.cidrsoperations/deployment-chartsmaster+24 -29
Customize query in gerrit

Related Objects
Search...

Event Timeline

JMeybohm triaged this task as Medium priority.Jul 27 2021, 4:00 PM
JMeybohm created this task.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 27 2021, 4:00 PM
JMeybohm updated the task description. (Show Details)Jul 27 2021, 4:00 PM
JMeybohm renamed this task from Allow to address Kubernets API servers from Calico NetworkPolicy to Allow to address Kubernets API servers from NetworkPolicy.Nov 12 2021, 2:38 PM
JMeybohm updated the task description. (Show Details)
JMeybohm added a subscriber: elukey.
JMeybohm updated the task description. (Show Details)Nov 24 2021, 6:52 AM
JMeybohm lowered the priority of this task from Medium to Low.Nov 18 2022, 4:02 PM
JMeybohm added a project: serviceops.
JMeybohm moved this task from Incoming 🐫 to ⎈Kubernetes on the serviceops board.
JMeybohm added a parent task: T307943: Update Kubernetes clusters to v1.23.
BTullis subscribed.Nov 18 2022, 4:04 PM
gerritbot added a comment.Mar 8 2023, 9:09 AM

Change 895696 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/deployment-charts@master] admin_ng/cert-manager: Remove dependency on kubernetesMasters.cidrs

https://gerrit.wikimedia.org/r/895696

gerritbot added a project: Patch-For-Review.Mar 8 2023, 9:09 AM
JMeybohm added a comment.EditedMar 8 2023, 9:20 AM

I've added a CR leveraging the service selector in calico network policies: https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/895696/
If that works, the other components using the manually defined kubernetesMasters.cidrs may follow:

  • helmfile_istio-gateways.yaml
  • flink-kubernetes-operator
  • knative-serving
  • kserve
  • rdf-streaming-updater (does not even use kubernetesMasters.cidrs, it has yet another copy of the master IPs)

Looking at the list is seems as if that network policy should be part of a helm chart module

JMeybohm mentioned this in T331894: Improve how we address outside k8s infrastructure from within charts (e.g. network policies).Mar 13 2023, 4:27 PM
jijiki claimed this task.Jul 4 2023, 1:41 PM
jijiki subscribed.
JMeybohm added a comment.Dec 19 2023, 11:53 AM

kube-state-metrics successfully introduced the pattern of using a calico networkpolicy with service selector to match masters in https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/974158

JMeybohm raised the priority of this task from Low to High.Apr 16 2024, 11:00 AM

We should prioritize T353464: Migrate wikikube control planes to hardware nodes because of T358936: Kubernetes apiserver probe failures on restart. Would be nice to have this done to lower configuration overhead, so raising this as well.

jijiki updated the task description. (Show Details)Wed, Apr 24, 11:05 AM
gerritbot added a comment.Mon, Apr 29, 10:41 AM

Change #1025296 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):

[operations/deployment-charts@master] admin_ng/helmfile_istio-gateway: Remove dependency on kubernetesMasters.cidrs

https://gerrit.wikimedia.org/r/1025296

gerritbot added a comment.Tue, Apr 30, 9:13 AM

Change #895696 merged by jenkins-bot:

[operations/deployment-charts@master] admin_ng/cert-manager: Remove dependency on kubernetesMasters.cidrs

https://gerrit.wikimedia.org/r/895696

gerritbot added a comment.Thu, May 2, 9:42 AM

Change #1026495 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):

[operations/deployment-charts@master] admin_ng/cert-manager: Remove dependency on kubernetesMasters.cidrs

https://gerrit.wikimedia.org/r/1026495

gerritbot added a comment.Thu, May 2, 10:37 AM

Change #1026495 merged by jenkins-bot:

[operations/deployment-charts@master] admin_ng/cert-manager: Remove dependency on kubernetesMasters.cidrs

https://gerrit.wikimedia.org/r/1026495

JMeybohm added subscribers: brouberol, klausman.Tue, May 7, 9:09 AM

I did deploy the cert-manager changes to aux, @brouberol did dse and @klausman will take care of ml clusters, thanks all

gerritbot added a comment.Wed, May 8, 1:16 PM

Change #1025296 merged by jenkins-bot:

[operations/deployment-charts@master] admin_ng/helmfile_istio-gateway: Remove dependency on kubernetesMasters.cidrs

https://gerrit.wikimedia.org/r/1025296

Maintenance_bot removed a project: Patch-For-Review.Wed, May 8, 1:31 PM
jijiki updated the task description. (Show Details)Wed, May 8, 2:27 PM
jijiki updated the task description. (Show Details)Wed, May 8, 2:32 PM
jijiki added a comment.Thu, May 9, 10:45 AM

Changes to istio-gateways in all clusters have been deployed

gerritbot added a comment.Thu, May 9, 2:13 PM

Change #1029573 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):

[operations/deployment-charts@master] (WIP) flink-kubernetes-operator: Remove dependency on kubernetesMasters.cidrs

https://gerrit.wikimedia.org/r/1029573

gerritbot added a project: Patch-For-Review.Thu, May 9, 2:13 PM
jijiki added a subscriber: dcausse.Tue, May 14, 1:50 PM

@dcausse and I will deploy 1029573 tomorrow EU morning

gerritbot added a comment.Wed, May 15, 8:18 AM

Change #1029573 merged by jenkins-bot:

[operations/deployment-charts@master] flink-kubernetes-operator: Remove dependency on kubernetesMasters.cidrs

https://gerrit.wikimedia.org/r/1029573

Maintenance_bot removed a project: Patch-For-Review.Wed, May 15, 8:31 AM
gerritbot added a comment.Wed, May 15, 8:38 AM

Change #1031810 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/deployment-charts@master] rdf-streaming-updater: Remove duplicate definition of k8s api-servers

https://gerrit.wikimedia.org/r/1031810

gerritbot added a project: Patch-For-Review.Wed, May 15, 8:38 AM
gerritbot added a comment.Wed, May 15, 8:45 AM

Change #1031811 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/deployment-charts@master] Remove kubernetesMasters definition from all wikikube values

https://gerrit.wikimedia.org/r/1031811

BTullis added a comment.Wed, May 15, 10:40 AM

I will have a look at the spark-operator too. Thanks all for working on this.

gmodena subscribed.Wed, May 15, 10:52 AM
Aklapper renamed this task from Allow to address Kubernets API servers from NetworkPolicy to Allow to address Kubernetes API servers from NetworkPolicy.Wed, May 15, 11:32 AM
gerritbot added a comment.Wed, May 15, 12:51 PM

Change #1031892 had a related patch set uploaded (by DCausse; author: DCausse):

[operations/deployment-charts@master] cirrus-streaming-updater: remove zk network policies

https://gerrit.wikimedia.org/r/1031892

jijiki updated the task description. (Show Details)Wed, May 15, 12:58 PM
jijiki added a comment.Wed, May 15, 1:05 PM
In T287491#9799054, @BTullis wrote:

I will have a look at the spark-operator too. Thanks all for working on this.

Send a review my way if you want, cheers

gerritbot added a comment.Wed, May 15, 1:19 PM

Change #1031901 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/deployment-charts@master] Remove kubernetesMasters definition from staging-codfw

https://gerrit.wikimedia.org/r/1031901

gerritbot added a comment.Wed, May 15, 1:23 PM

Change #1031901 merged by JMeybohm:

[operations/deployment-charts@master] Remove kubernetesMasters definition from staging-codfw

https://gerrit.wikimedia.org/r/1031901

BTullis added a project: Data-Platform-SRE (2024.05.06 - 2024.05.26).Wed, May 15, 1:56 PM
BTullis moved this task from Backlog to In Progress on the Data-Platform-SRE (2024.05.06 - 2024.05.26) board.
gerritbot added a comment.Wed, May 15, 2:08 PM

Change #1031593 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/deployment-charts@master] Remove kubernetesMasters definition from dse-k8s

https://gerrit.wikimedia.org/r/1031593

BTullis added a comment.Wed, May 15, 2:15 PM
In T287491#9799751, @jijiki wrote:
In T287491#9799054, @BTullis wrote:

I will have a look at the spark-operator too. Thanks all for working on this.

Send a review my way if you want, cheers

It turns out that this was simpler than expected because spark-operator is already using calico network policies to access the k8s API.

Now that the flink-operator has been updated, that means I think we can just remove the unused values: https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1031593

BTullis moved this task from In Progress to Needs Review on the Data-Platform-SRE (2024.05.06 - 2024.05.26) board.Wed, May 15, 2:29 PM
gerritbot added a comment.Thu, May 16, 8:25 AM

Change #1031810 merged by jenkins-bot:

[operations/deployment-charts@master] rdf-streaming-updater: Remove duplicate definition of k8s and zk

https://gerrit.wikimedia.org/r/1031810

gerritbot added a comment.Thu, May 16, 8:31 AM

Change #1031593 merged by Btullis:

[operations/deployment-charts@master] Remove kubernetesMasters definition from dse-k8s

https://gerrit.wikimedia.org/r/1031593

BTullis updated the task description. (Show Details)Thu, May 16, 8:32 AM
gerritbot added a comment.Thu, May 16, 8:45 AM

Change #1031811 merged by jenkins-bot:

[operations/deployment-charts@master] Remove kubernetesMasters definition from all wikikube values

https://gerrit.wikimedia.org/r/1031811

jijiki added a comment.Thu, May 16, 9:24 AM

@klausman you can go head with kserve and knative-serving when you have some time, ping me for reviews etc

klausman added a comment.Thu, May 16, 9:30 AM
In T287491#9803609, @jijiki wrote:

@klausman you can go head with kserve and knative-serving when you have some time, ping me for reviews etc

Will do! I'll likely get to it next week.

BTullis moved this task from Needs Review to Done on the Data-Platform-SRE (2024.05.06 - 2024.05.26) board.Fri, May 17, 9:58 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL