Page MenuHomePhabricator
Create Task
Maniphest T359423

Migrate charts to Calico Network Policies
Open, MediumPublic
Actions

Description

The following charts will need to be migrated to the base.networkpolicy.egress.external-services helm template, rendering Calico network policies:

  • benthos-cache-invalidator (also needs securityContext update: T362978) @kamila
  • changeprop
  • datahub @BTullis @brouberol
  • eventgate (also needs mesh.configuration and securityContext update: T346638, T362978) @JMeybohm
    • eventgate-analytics
    • eventgate-analytics-external
    • eventgate-logging-external
    • eventgate-main
  • eventstreams @Scott_French (https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1037870)
    • eventstreams
    • eventstreams-internal
  • flink-app @bking , see T373195
    • cirrus-streaming-updater
    • mw-page-content-change-enrich
    • rdf-streaming-updater
  • flink-operator @bking , see T373195
  • mediawiki (also needs securityContext update: T362978)
    • mw-debug
    • all other mediawiki releases
  • tegola-vector-tiles
  • spark-history @brouberol
  • kserve-inference (also needs securityContext update: T362978) @klausman
  • AQS 2.0 services T364533
    • device-analytics (cassandra aqs)
    • edit-analytics (cassandra aqs, druid-public)
    • editor-analytics (cassandra aqs, druid-public)
    • geo-analytics (cassandra aqs)
    • media-analytics (cassandra aqs)
    • page-analytics (cassandra aqs)
  • image-suggestions (cassandra-aqs) T364533
  • OpenTelemetryCollector

Details

SubjectRepoBranchLines +/-
cfssl-issuer: Add external-services supportoperations/deployment-chartsmaster+33 -2
global_config: Add pki::multirootca IPs to external-servicesoperations/puppetproduction+17 -4
eventstreams: adopt base.external-services-networkpolicyoperations/deployment-chartsmaster+74 -32
datahub: replace IPs by Services in network policiesoperations/deployment-chartsmaster+30 -43
datahub-next: restore IP-based networkpolicy to datahubsearchoperations/deployment-chartsmaster+10 -3
datahub: fix label matching beetween pods and networkpoliciesoperations/deployment-chartsmaster+8 -8
datahub: update datahubsearch hostname to use external-servicesoperations/deployment-chartsmaster+1 -1
Deploy calico network policy templates to all datahub chartsoperations/deployment-chartsmaster+257 -8
datahub-next: replace IPs by Services in network policiesoperations/deployment-chartsmaster+19 -38
Migrate AQS2 services and image-suggestions to calico network policiesoperations/deployment-chartsmaster+181 -323
benthos: adopt securityContext and base.external-services-networkpolicyoperations/deployment-chartsmaster+82 -26
eventgate-*: Migrate to base.external-services-networkpolicyoperations/deployment-chartsmaster+80 -47
eventgate: Update mesh modulesoperations/deployment-chartsmaster+123 -73
charts/kserve-inference: Wire up generated network policy for LW servicesoperations/deployment-chartsmaster+119 -3
changeprop-jobqueue: Migrate to base.external-services-networkpolicyoperations/deployment-chartsmaster+5 -185
changeprop: Migrate to base.external-services-networkpolicyoperations/deployment-chartsmaster+5 -184
changeprop: Add base.external-services-networkpolicy:1.0operations/deployment-chartsmaster+63 -5
changeprop: Update mesh modulesoperations/deployment-chartsmaster+123 -73
Make datahub networkpolicy include/template consistentoperations/deployment-chartsmaster+3 -3
Fix whitespace in the external-services values for datahub stagingoperations/deployment-chartsmaster+1 -1
spark-history: bypass Kerberos principal hostname reverse DNS check for namenodeoperations/deployment-chartsmaster+7 -1
spark-history: fix egress network policiesoperations/deployment-chartsmaster+12 -4
spark-history: replace hardcoded CIDRs by service names to generate egress policiesoperations/deployment-chartsmaster+7 -70
spark-history: add external-services egress network policy templateoperations/deployment-chartsmaster+49 -3
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
brouberol updated the task description. (Show Details)Mar 25 2024, 12:37 PM
gerritbot added a comment.Mar 25 2024, 12:44 PM

Change #1013989 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/deployment-charts@master] spark-history: add external-services egress network policy template

https://gerrit.wikimedia.org/r/1013989

gerritbot added a project: Patch-For-Review.Mar 25 2024, 12:44 PM

Change #1013990 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/deployment-charts@master] spark-history: replace hardcoded CIDRs by service names to generate egress policies

https://gerrit.wikimedia.org/r/1013990

gerritbot added a comment.Mar 25 2024, 12:57 PM

Change #1013989 merged by Brouberol:

[operations/deployment-charts@master] spark-history: add external-services egress network policy template

https://gerrit.wikimedia.org/r/1013989

gerritbot added a comment.Mar 25 2024, 1:06 PM

Change #1013990 merged by Brouberol:

[operations/deployment-charts@master] spark-history: replace hardcoded CIDRs by service names to generate egress policies

https://gerrit.wikimedia.org/r/1013990

gerritbot added a comment.Mar 25 2024, 1:11 PM

Change #1013997 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/deployment-charts@master] spark-history: fix egress network policies

https://gerrit.wikimedia.org/r/1013997

gerritbot added a comment.Mar 25 2024, 1:22 PM

Change #1013997 merged by Brouberol:

[operations/deployment-charts@master] spark-history: fix egress network policies

https://gerrit.wikimedia.org/r/1013997

gerritbot added a comment.Mar 25 2024, 1:51 PM

Change #1014010 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/deployment-charts@master] spark-history: bypass Kerberos principal hostname reverse DNS check for namenode

https://gerrit.wikimedia.org/r/1014010

gerritbot added a comment.Mar 26 2024, 10:26 AM

Change #1014010 merged by Brouberol:

[operations/deployment-charts@master] spark-history: bypass Kerberos principal hostname reverse DNS check for namenode

https://gerrit.wikimedia.org/r/1014010

gerritbot added a comment.Mar 26 2024, 3:48 PM

Change #1014538 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/deployment-charts@master] changeprop: Update mesh modules

https://gerrit.wikimedia.org/r/1014538

gerritbot added a comment.Mar 26 2024, 3:48 PM

Change #1014539 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/deployment-charts@master] changeprop: Add base.external-services-networkpolicy:1.0

https://gerrit.wikimedia.org/r/1014539

gerritbot added a comment.Mar 26 2024, 3:48 PM

Change #1014540 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/deployment-charts@master] changeprop: Migrate to base.external-services-networkpolicy

https://gerrit.wikimedia.org/r/1014540

gerritbot added a comment.Mar 26 2024, 3:52 PM

Change #1014542 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/deployment-charts@master] changeprop-jobqueue: Migrate to base.external-services-networkpolicy

https://gerrit.wikimedia.org/r/1014542

JMeybohm updated the task description. (Show Details)Mar 26 2024, 4:03 PM
brouberol updated the task description. (Show Details)Mar 26 2024, 4:06 PM
BTullis updated the task description. (Show Details)Mar 26 2024, 10:26 PM
gerritbot added a comment.Mar 26 2024, 10:27 PM

Change #1014646 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/deployment-charts@master] Fix typo in the external-services values for datahub staging

https://gerrit.wikimedia.org/r/1014646

BTullis added a comment.EditedMar 26 2024, 10:30 PM

I needed to start work on the DataHub migration because SSO broke due to the switch of IDP servers: https://sal.toolforge.org/log/UMCPdY4BhuQtenzvi5Pe

Slack thread here.

CR here: https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1014065

So far, I have only migrated the CAS IDP services, but there are still several others that can be migrated too, including:

  • datahubsearch
  • mariadb (analytics_meta)

We can probably remove the LDAP policies from this deployment, now that CAS is enabled. Otherwise I would migrate that too.

gerritbot added a comment.Mar 26 2024, 10:54 PM

Change #1014646 merged by jenkins-bot:

[operations/deployment-charts@master] Fix whitespace in the external-services values for datahub staging

https://gerrit.wikimedia.org/r/1014646

BTullis updated the task description. (Show Details)Mar 26 2024, 10:58 PM
BTullis updated the task description. (Show Details)Mar 26 2024, 11:08 PM
gerritbot added a comment.Mar 27 2024, 10:52 AM

Change #1014652 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/deployment-charts@master] Make datahub networkpolicy include/template consistent

https://gerrit.wikimedia.org/r/1014652

brouberol updated the task description. (Show Details)Mar 27 2024, 12:50 PM
brouberol added a subscriber: klausman.
gerritbot added a comment.Mar 27 2024, 12:51 PM

Change #1015029 had a related patch set uploaded (by Klausman; author: Klausman):

[operations/deployment-charts@master] charts/kserve-inference: Wire up generated network policy for LW services

https://gerrit.wikimedia.org/r/1015029

gerritbot added a comment.Mar 27 2024, 1:23 PM

Change #1014652 merged by jenkins-bot:

[operations/deployment-charts@master] Make datahub networkpolicy include/template consistent

https://gerrit.wikimedia.org/r/1014652

kamila updated the task description. (Show Details)Mar 27 2024, 2:43 PM
kamila subscribed.
gerritbot added a comment.Mar 27 2024, 3:37 PM

Change #1014538 merged by jenkins-bot:

[operations/deployment-charts@master] changeprop: Update mesh modules

https://gerrit.wikimedia.org/r/1014538

gerritbot added a comment.Mar 27 2024, 3:38 PM

Change #1014539 merged by jenkins-bot:

[operations/deployment-charts@master] changeprop: Add base.external-services-networkpolicy:1.0

https://gerrit.wikimedia.org/r/1014539

gerritbot added a comment.Mar 27 2024, 3:38 PM

Change #1014540 merged by jenkins-bot:

[operations/deployment-charts@master] changeprop: Migrate to base.external-services-networkpolicy

https://gerrit.wikimedia.org/r/1014540

gerritbot added a comment.Mar 27 2024, 4:03 PM

Change #1014542 merged by jenkins-bot:

[operations/deployment-charts@master] changeprop-jobqueue: Migrate to base.external-services-networkpolicy

https://gerrit.wikimedia.org/r/1014542

JMeybohm updated the task description. (Show Details)Mar 27 2024, 4:16 PM
BTullis mentioned this in T360531: [Commons Impact Metrics] Refactor/create helm config for AQS service accessing both Cassandra and Druid.Mar 28 2024, 11:06 AM
gerritbot added a comment.Mar 28 2024, 11:28 AM

Change #1015029 merged by jenkins-bot:

[operations/deployment-charts@master] charts/kserve-inference: Wire up generated network policy for LW services

https://gerrit.wikimedia.org/r/1015029

JMeybohm updated the task description. (Show Details)Apr 10 2024, 12:44 PM

Grouped the todo list by chart, some of those also need mesh.configuration updates due to T346638: Rename the envoy's uses_ingress option to sets_sni which could be bundled with this

gerritbot added a comment.Apr 11 2024, 11:40 AM

Change #1019007 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/deployment-charts@master] eventgate: Update mesh modules

https://gerrit.wikimedia.org/r/1019007

gerritbot added a comment.Apr 11 2024, 12:02 PM

Change #1019018 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/deployment-charts@master] eventgate-*: Migrate to base.external-services-networkpolicy

https://gerrit.wikimedia.org/r/1019018

JMeybohm updated the task description. (Show Details)Apr 12 2024, 7:32 AM
JMeybohm updated the task description. (Show Details)Apr 23 2024, 10:15 AM
gerritbot added a comment.Apr 25 2024, 8:06 AM

Change #1019007 merged by jenkins-bot:

[operations/deployment-charts@master] eventgate: Update mesh modules

https://gerrit.wikimedia.org/r/1019007

gerritbot added a comment.Apr 25 2024, 8:06 AM

Change #1019018 merged by jenkins-bot:

[operations/deployment-charts@master] eventgate-*: Migrate to base.external-services-networkpolicy

https://gerrit.wikimedia.org/r/1019018

JMeybohm updated the task description. (Show Details)Apr 25 2024, 9:34 AM
Maintenance_bot removed a project: Patch-For-Review.Apr 26 2024, 6:54 PM
BTullis updated the task description. (Show Details)May 9 2024, 10:18 AM
BTullis updated the task description. (Show Details)
BTullis updated the task description. (Show Details)May 9 2024, 10:50 AM
gerritbot added a comment.May 9 2024, 6:27 PM

Change #1028910 had a related patch set uploaded (by Scott French; author: Scott French):

[operations/deployment-charts@master] benthos: adopt securityContext and base.external-services-networkpolicy

https://gerrit.wikimedia.org/r/1028910

gerritbot added a project: Patch-For-Review.May 9 2024, 6:27 PM
gerritbot added a comment.May 14 2024, 3:43 PM

Change #1028910 merged by jenkins-bot:

[operations/deployment-charts@master] benthos: adopt securityContext and base.external-services-networkpolicy

https://gerrit.wikimedia.org/r/1028910

Maintenance_bot removed a project: Patch-For-Review.May 14 2024, 4:32 PM
gerritbot added a comment.May 22 2024, 11:46 AM

Change #1033405 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/deployment-charts@master] Migrate AQS2 services and image-suggestions to calico network policies

https://gerrit.wikimedia.org/r/1033405

gerritbot added a project: Patch-For-Review.May 22 2024, 11:46 AM
gerritbot added a comment.May 24 2024, 11:05 AM

Change #1033405 merged by jenkins-bot:

[operations/deployment-charts@master] Migrate AQS2 services and image-suggestions to calico network policies

https://gerrit.wikimedia.org/r/1033405

Maintenance_bot removed a project: Patch-For-Review.May 24 2024, 11:31 AM
Scott_French updated the task description. (Show Details)May 31 2024, 10:48 PM
gerritbot added a comment.Jun 1 2024, 12:23 AM

Change #1037870 had a related patch set uploaded (by Scott French; author: Scott French):

[operations/deployment-charts@master] eventstreams: adopt base.external-services-networkpolicy

https://gerrit.wikimedia.org/r/1037870

gerritbot added a project: Patch-For-Review.Jun 1 2024, 12:23 AM
BTullis closed subtask T364533: Migrate AQS and image-suggestions services to Calico Network Policies as Resolved.Jun 4 2024, 4:08 PM
JMeybohm updated the task description. (Show Details)Jun 7 2024, 3:38 PM
brouberol updated the task description. (Show Details)Jun 10 2024, 6:57 AM
gerritbot added a comment.Jun 10 2024, 7:29 AM

Change #1040874 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/deployment-charts@master] datahub-next: replace IPs by Services in network policies

https://gerrit.wikimedia.org/r/1040874

gerritbot added a comment.Jun 10 2024, 7:29 AM

Change #1040875 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/deployment-charts@master] datahub: replace IPs by Services in network policies

https://gerrit.wikimedia.org/r/1040875

brouberol updated the task description. (Show Details)Jun 10 2024, 7:38 AM
gerritbot added a comment.Jun 11 2024, 9:55 AM

Change #1040874 merged by Brouberol:

[operations/deployment-charts@master] datahub-next: replace IPs by Services in network policies

https://gerrit.wikimedia.org/r/1040874

gerritbot added a comment.Jun 11 2024, 2:31 PM

Change #1041671 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/deployment-charts@master] datahub: update datahubsearch hostname to use external-services

https://gerrit.wikimedia.org/r/1041671

gerritbot added a comment.Jun 11 2024, 2:54 PM

Change #1041676 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/deployment-charts@master] Deploy calico network policy templates to all datahub charts

https://gerrit.wikimedia.org/r/1041676

JMeybohm updated the task description. (Show Details)Jun 12 2024, 1:21 PM
gerritbot added a comment.Jun 13 2024, 8:57 AM

Change #1041676 merged by Brouberol:

[operations/deployment-charts@master] Deploy calico network policy templates to all datahub charts

https://gerrit.wikimedia.org/r/1041676

gerritbot added a comment.Jun 13 2024, 9:16 AM

Change #1042952 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/deployment-charts@master] datahub-next: restore IP-based networkpolicy to datahubsearch

https://gerrit.wikimedia.org/r/1042952

gerritbot added a comment.Jun 13 2024, 9:17 AM

Change #1041671 abandoned by Brouberol:

[operations/deployment-charts@master] datahub: update datahubsearch hostname to use external-services

Reason:

https://gerrit.wikimedia.org/r/1041671

gerritbot added a comment.Jun 13 2024, 9:32 AM

Change #1042964 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/deployment-charts@master] datahub: fix label matching beetween pods and networkpolicies

https://gerrit.wikimedia.org/r/1042964

gerritbot added a comment.Jun 13 2024, 9:50 AM

Change #1042964 merged by jenkins-bot:

[operations/deployment-charts@master] datahub: fix label matching beetween pods and networkpolicies

https://gerrit.wikimedia.org/r/1042964

gerritbot added a comment.Jun 13 2024, 9:51 AM

Change #1042952 merged by jenkins-bot:

[operations/deployment-charts@master] datahub-next: restore IP-based networkpolicy to datahubsearch

https://gerrit.wikimedia.org/r/1042952

gerritbot added a comment.Jun 13 2024, 10:37 AM

Change #1040875 merged by Brouberol:

[operations/deployment-charts@master] datahub: replace IPs by Services in network policies

https://gerrit.wikimedia.org/r/1040875

akosiaris updated the task description. (Show Details)Jul 9 2024, 1:55 PM
JMeybohm updated the task description. (Show Details)Aug 22 2024, 3:26 PM
JMeybohm added a subscriber: Scott_French.
brouberol updated the task description. (Show Details)Aug 22 2024, 3:37 PM
gerritbot added a comment.Aug 26 2024, 5:33 PM

Change #1037870 merged by jenkins-bot:

[operations/deployment-charts@master] eventstreams: adopt base.external-services-networkpolicy

https://gerrit.wikimedia.org/r/1037870

Scott_French updated the task description. (Show Details)Aug 26 2024, 6:16 PM
klausman updated the task description. (Show Details)Aug 27 2024, 9:03 AM
klausman updated the task description. (Show Details)Aug 27 2024, 9:11 AM
akosiaris updated the task description. (Show Details)Aug 27 2024, 4:16 PM
BTullis updated the task description. (Show Details)Aug 27 2024, 5:07 PM
gerritbot added a comment.Aug 29 2024, 12:37 PM

Change #1068754 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/puppet@production] global_config: Add pki::multirootca IPs to external-services

https://gerrit.wikimedia.org/r/1068754

gerritbot added a comment.Aug 29 2024, 12:40 PM

Change #1068768 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/deployment-charts@master] cfssl-issuer: Add external-services support

https://gerrit.wikimedia.org/r/1068768

gerritbot added a comment.Aug 30 2024, 6:56 AM

Change #1068754 merged by JMeybohm:

[operations/puppet@production] global_config: Add pki::multirootca IPs to external-services

https://gerrit.wikimedia.org/r/1068754

gerritbot added a comment.Sep 2 2024, 2:39 PM

Change #1068768 merged by jenkins-bot:

[operations/deployment-charts@master] cfssl-issuer: Add external-services support

https://gerrit.wikimedia.org/r/1068768

Maintenance_bot removed a project: Patch-For-Review.Sep 2 2024, 3:31 PM
bking updated the task description. (Show Details)Sep 9 2024, 5:37 PM
bking updated the task description. (Show Details)Sep 24 2024, 1:47 PM
gmodena subscribed.Sep 25 2024, 11:01 AM
bking closed subtask T373195: Migrate Search Platform-owned helm charts to Calico Network Policies as Resolved.Nov 8 2024, 5:12 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits