Page MenuHomePhabricator
Create Task
Maniphest T331858

Deploy spark-operator webhook
Closed, ResolvedPublic1 Estimated Story Points
Actions

Description

In order to be able to deploy SparkApplication relying on volumes, hadoopConfigMap, sparkConfigMap, init containers and so on we need to enable the spark-operator webhook which will mutate spark pods at creation
See for more details on what component are mutated on pods by the webhook: https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/blob/master/pkg/webhook/webhook.go#L539

  • Add webhook
  • Rely on cert-manager instead of gencert.sh mechanism
  • Authorize access from API server to webhook

Done is

  • webhook is enabled on spark operator and registered on K8S cluster
  • spark pods are well mutated when adding hadoopConfigMap

Details

SubjectRepoBranchLines +/-
spark: udapte networkpolicy to authorize kubernetes-api to contact webhook serviceoperations/deployment-chartsmaster+8 -8
spark: add webhook rights to ClusterRoleBindingoperations/deployment-chartsmaster+5 -5
spark-operator: enable spark operator mutation webhookoperations/deployment-chartsmaster+211 -26
Customize query in gerrit

Event Timeline

• nfraison created this task.Mar 13 2023, 10:24 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 13 2023, 10:24 AM
• nfraison changed the task status from Open to In Progress.Mar 13 2023, 10:25 AM
• nfraison claimed this task.
• nfraison updated the task description. (Show Details)Mar 13 2023, 10:50 AM

Ex. of Networkpolicy:
https://gerrit.wikimedia.org/r/plugins/gitiles/operations/deployment-charts/+/refs/heads/master/charts/kserve/templates/networkpolicy.yaml

gerritbot added a comment.Mar 13 2023, 2:20 PM

Change 897895 had a related patch set uploaded (by Nicolas Fraison; author: Nicolas Fraison):

[operations/deployment-charts@master] spark-operator: enable spark operator mutation webhook

https://gerrit.wikimedia.org/r/897895

gerritbot added a project: Patch-For-Review.Mar 13 2023, 2:20 PM
JArguello-WMF moved this task from Backlog to To be discussed on the Shared-Data-Infrastructure board.Mar 14 2023, 3:52 PM
JArguello-WMF set the point value for this task to 1.Mar 14 2023, 5:29 PM
JArguello-WMF moved this task from To be discussed to 2022-23 Q4 Wrap up on the Shared-Data-Infrastructure board.
JArguello-WMF edited projects, added Shared-Data-Infrastructure (2022-23 Q4 Wrap up); removed Shared-Data-Infrastructure.
JArguello-WMF triaged this task as High priority.Mar 14 2023, 6:00 PM
• nfraison moved this task from Next Up to In Progress on the Shared-Data-Infrastructure (2022-23 Q4 Wrap up) board.Mar 15 2023, 9:03 AM
• nfraison moved this task from In Progress to In Review on the Shared-Data-Infrastructure (2022-23 Q4 Wrap up) board.
• nfraison updated the task description. (Show Details)Mar 15 2023, 9:06 AM
JArguello-WMF moved this task from In Review to Ready to Deploy on the Shared-Data-Infrastructure (2022-23 Q4 Wrap up) board.Mar 20 2023, 1:37 PM
gerritbot added a comment.Mar 21 2023, 10:13 AM

Change 897895 merged by Nicolas Fraison:

[operations/deployment-charts@master] spark-operator: enable spark operator mutation webhook

https://gerrit.wikimedia.org/r/897895

Maintenance_bot removed a project: Patch-For-Review.Mar 21 2023, 10:32 AM
gerritbot added a comment.Mar 21 2023, 1:28 PM

Change 901581 had a related patch set uploaded (by Nicolas Fraison; author: Nicolas Fraison):

[operations/deployment-charts@master] spark: add webhook rights to ClusterRoleBinding

https://gerrit.wikimedia.org/r/901581

gerritbot added a project: Patch-For-Review.Mar 21 2023, 1:28 PM
gerritbot added a comment.Mar 21 2023, 1:34 PM

Change 901581 merged by Nicolas Fraison:

[operations/deployment-charts@master] spark: add webhook rights to ClusterRoleBinding

https://gerrit.wikimedia.org/r/901581

Maintenance_bot removed a project: Patch-For-Review.Mar 21 2023, 2:11 PM
gerritbot added a comment.Mar 21 2023, 3:57 PM

Change 901618 had a related patch set uploaded (by Nicolas Fraison; author: Nicolas Fraison):

[operations/deployment-charts@master] spark: udapte networkpolicy to authorize kubernetes-api to contact webhook service

https://gerrit.wikimedia.org/r/901618

gerritbot added a project: Patch-For-Review.Mar 21 2023, 3:57 PM
gerritbot added a comment.Mar 21 2023, 4:27 PM

Change 901618 merged by Nicolas Fraison:

[operations/deployment-charts@master] spark: udapte networkpolicy to authorize kubernetes-api to contact webhook service

https://gerrit.wikimedia.org/r/901618

Maintenance_bot removed a project: Patch-For-Review.Mar 21 2023, 4:30 PM
• nfraison moved this task from Ready to Deploy to Done on the Shared-Data-Infrastructure (2022-23 Q4 Wrap up) board.Mar 22 2023, 10:25 AM
JArguello-WMF closed this task as Resolved.Mar 31 2023, 1:41 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL