Page MenuHomePhabricator

Migrate charts to Calico Network Policies
Open, MediumPublic

Description

The following charts will need to be migrated to the base.networkpolicy.egress.external-services helm template, rendering Calico network policies:

  • benthos-cache-invalidator (also needs securityContext update: T362978) @kamila
  • changeprop
  • datahub (also needs mesh.configuration and securityContext update: T346638, T362978) @BTullis
  • eventgate (also needs mesh.configuration and securityContext update: T346638, T362978) @JMeybohm
    • eventgate-analytics
    • eventgate-analytics-external
    • eventgate-logging-external
    • eventgate-main
  • eventstreams (also needs securityContext update: T362978)
    • eventstreams
    • eventstreams-internal
  • flink-app (also needs mesh.configuration and securityContext update: T346638, T362978)
    • cirrus-streaming-updater
    • mw-page-content-change-enrich
    • rdf-streaming-updater
  • flink-operator (also needs securityContext update: T362978)
  • mediawiki (also needs securityContext update: T362978)
    • mw-debug
    • all other mediawiki releases
  • tegola-vector-tiles (also needs mesh.configuration and securityContext update: T346638, T362978)
  • spark-history @brouberol
  • kserve-inference (also needs securityContext update: T362978) @klausman
  • AQS 2.0 services T364533
    • device-analytics (cassandra aqs)
    • edit-analytics (cassandra aqs, druid-public)
    • editor-analytics (cassandra aqs, druid-public)
    • geo-analytics (cassandra aqs)
    • media-analytics (cassandra aqs)
    • page-analytics (cassandra aqs)
  • image-suggestions (cassandra-aqs) T364533

Details

SubjectRepoBranchLines +/-
operations/deployment-chartsmaster+181 -323
operations/deployment-chartsmaster+82 -26
operations/deployment-chartsmaster+80 -47
operations/deployment-chartsmaster+123 -73
operations/deployment-chartsmaster+119 -3
operations/deployment-chartsmaster+5 -185
operations/deployment-chartsmaster+5 -184
operations/deployment-chartsmaster+63 -5
operations/deployment-chartsmaster+123 -73
operations/deployment-chartsmaster+3 -3
operations/deployment-chartsmaster+1 -1
operations/deployment-chartsmaster+7 -1
operations/deployment-chartsmaster+12 -4
operations/deployment-chartsmaster+7 -70
operations/deployment-chartsmaster+49 -3
Show related patches Customize query in gerrit

Event Timeline

Gehel triaged this task as Medium priority.Mar 20 2024, 9:04 AM
Gehel moved this task from Incoming to Toil / Automation on the Data-Platform-SRE board.

Change #1013989 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/deployment-charts@master] spark-history: add external-services egress network policy template

https://gerrit.wikimedia.org/r/1013989

Change #1013990 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/deployment-charts@master] spark-history: replace hardcoded CIDRs by service names to generate egress policies

https://gerrit.wikimedia.org/r/1013990

Change #1013989 merged by Brouberol:

[operations/deployment-charts@master] spark-history: add external-services egress network policy template

https://gerrit.wikimedia.org/r/1013989

Change #1013990 merged by Brouberol:

[operations/deployment-charts@master] spark-history: replace hardcoded CIDRs by service names to generate egress policies

https://gerrit.wikimedia.org/r/1013990

Change #1013997 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/deployment-charts@master] spark-history: fix egress network policies

https://gerrit.wikimedia.org/r/1013997

Change #1013997 merged by Brouberol:

[operations/deployment-charts@master] spark-history: fix egress network policies

https://gerrit.wikimedia.org/r/1013997

Change #1014010 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/deployment-charts@master] spark-history: bypass Kerberos principal hostname reverse DNS check for namenode

https://gerrit.wikimedia.org/r/1014010

Change #1014010 merged by Brouberol:

[operations/deployment-charts@master] spark-history: bypass Kerberos principal hostname reverse DNS check for namenode

https://gerrit.wikimedia.org/r/1014010

Change #1014538 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/deployment-charts@master] changeprop: Update mesh modules

https://gerrit.wikimedia.org/r/1014538

Change #1014539 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/deployment-charts@master] changeprop: Add base.external-services-networkpolicy:1.0

https://gerrit.wikimedia.org/r/1014539

Change #1014540 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/deployment-charts@master] changeprop: Migrate to base.external-services-networkpolicy

https://gerrit.wikimedia.org/r/1014540

Change #1014542 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/deployment-charts@master] changeprop-jobqueue: Migrate to base.external-services-networkpolicy

https://gerrit.wikimedia.org/r/1014542

Change #1014646 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/deployment-charts@master] Fix typo in the external-services values for datahub staging

https://gerrit.wikimedia.org/r/1014646

I needed to start work on the DataHub migration because SSO broke due to the switch of IDP servers: https://sal.toolforge.org/log/UMCPdY4BhuQtenzvi5Pe

Slack thread here.

CR here: https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1014065

So far, I have only migrated the CAS IDP services, but there are still several others that can be migrated too, including:

  • datahubsearch
  • mariadb (analytics_meta)

We can probably remove the LDAP policies from this deployment, now that CAS is enabled. Otherwise I would migrate that too.

Change #1014646 merged by jenkins-bot:

[operations/deployment-charts@master] Fix whitespace in the external-services values for datahub staging

https://gerrit.wikimedia.org/r/1014646

Change #1014652 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/deployment-charts@master] Make datahub networkpolicy include/template consistent

https://gerrit.wikimedia.org/r/1014652

Change #1015029 had a related patch set uploaded (by Klausman; author: Klausman):

[operations/deployment-charts@master] charts/kserve-inference: Wire up generated network policy for LW services

https://gerrit.wikimedia.org/r/1015029

Change #1014652 merged by jenkins-bot:

[operations/deployment-charts@master] Make datahub networkpolicy include/template consistent

https://gerrit.wikimedia.org/r/1014652

Change #1014538 merged by jenkins-bot:

[operations/deployment-charts@master] changeprop: Update mesh modules

https://gerrit.wikimedia.org/r/1014538

Change #1014539 merged by jenkins-bot:

[operations/deployment-charts@master] changeprop: Add base.external-services-networkpolicy:1.0

https://gerrit.wikimedia.org/r/1014539

Change #1014540 merged by jenkins-bot:

[operations/deployment-charts@master] changeprop: Migrate to base.external-services-networkpolicy

https://gerrit.wikimedia.org/r/1014540

Change #1014542 merged by jenkins-bot:

[operations/deployment-charts@master] changeprop-jobqueue: Migrate to base.external-services-networkpolicy

https://gerrit.wikimedia.org/r/1014542

Change #1015029 merged by jenkins-bot:

[operations/deployment-charts@master] charts/kserve-inference: Wire up generated network policy for LW services

https://gerrit.wikimedia.org/r/1015029

Grouped the todo list by chart, some of those also need mesh.configuration updates due to T346638: Rename the envoy's uses_ingress option to sets_sni which could be bundled with this

Change #1019007 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/deployment-charts@master] eventgate: Update mesh modules

https://gerrit.wikimedia.org/r/1019007

Change #1019018 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/deployment-charts@master] eventgate-*: Migrate to base.external-services-networkpolicy

https://gerrit.wikimedia.org/r/1019018

Change #1019007 merged by jenkins-bot:

[operations/deployment-charts@master] eventgate: Update mesh modules

https://gerrit.wikimedia.org/r/1019007

Change #1019018 merged by jenkins-bot:

[operations/deployment-charts@master] eventgate-*: Migrate to base.external-services-networkpolicy

https://gerrit.wikimedia.org/r/1019018

BTullis updated the task description. (Show Details)

Change #1028910 had a related patch set uploaded (by Scott French; author: Scott French):

[operations/deployment-charts@master] benthos: adopt securityContext and base.external-services-networkpolicy

https://gerrit.wikimedia.org/r/1028910

Change #1028910 merged by jenkins-bot:

[operations/deployment-charts@master] benthos: adopt securityContext and base.external-services-networkpolicy

https://gerrit.wikimedia.org/r/1028910

Change #1033405 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/deployment-charts@master] Migrate AQS2 services and image-suggestions to calico network policies

https://gerrit.wikimedia.org/r/1033405

Change #1033405 merged by jenkins-bot:

[operations/deployment-charts@master] Migrate AQS2 services and image-suggestions to calico network policies

https://gerrit.wikimedia.org/r/1033405