Page MenuHomePhabricator
Create Task
Maniphest T331526

eventutilities-python should support using Kafka TLS ports
Closed, ResolvedPublic
Actions

Description

Currently, eventutilities-python only supports configuration of the Kafka servers to use via the KAFKA_BOOTSTRAP_SERVERS env var.

Before deploying in wikikube, we should be able to use the Kafka TLS 9093 port. To do this, we need to set a couple of other kafka client properties:

For librdkafka clients, these properties are:

security.protocol: ssl
ssl.ca.location: /etc/ssl/certs/wmf-ca-certificates.crt

We'll need to set the correpsonding configs for the Java Kafka client via the Flink connector. These will need to be passed through to the Kafka Source and Sink builders via setProperties.

  • eventutilities-python supports setting other Kafka client properties
  • mediawiki-event-enrichment supports setting other Kafka client properties via eventutilities-python
  • mediawiki-page-content-change-enrichment uses Kafka TLS ports in DSE

Parameterization improvements are also tracked in T328478: Refactor parameterization of eventutilities-python and mediawiki-event-enrichment

Details

SubjectRepoBranchLines +/-
dse-k8s-services/mediawiki-page-content-change-enrichment - use kafka SSLoperations/deployment-chartsmaster+2 -2
mediawiki-page-content-change-enrichment: use kafka TLSoperations/deployment-chartsmaster+9 -4
mediawiki-page-content-change-enrichment - use kafka 9092 until we can use TLSoperations/deployment-chartsmaster+1 -1
Customize query in gerrit

Related Objects
Search...

Event Timeline

Ottomata created this task.Mar 8 2023, 12:41 PM
Restricted Application added a project: Data-Engineering. · View Herald TranscriptMar 8 2023, 12:41 PM
Ottomata added a parent task: T328478: Refactor parameterization of eventutilities-python and mediawiki-event-enrichment.Mar 8 2023, 12:41 PM
gerritbot added a comment.Mar 8 2023, 12:43 PM

Change 895754 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/deployment-charts@master] mediawiki-page-content-change-enrichment - use kafka 9092 until we can use TLS

https://gerrit.wikimedia.org/r/895754

gerritbot added a project: Patch-For-Review.Mar 8 2023, 12:43 PM
gerritbot added a comment.Mar 8 2023, 12:46 PM

Change 895754 merged by Ottomata:

[operations/deployment-charts@master] mediawiki-page-content-change-enrichment - use kafka 9092 until we can use TLS

https://gerrit.wikimedia.org/r/895754

Maintenance_bot removed a project: Patch-For-Review.Mar 8 2023, 1:10 PM
Ottomata moved this task from Backlog to To be Estimated/To be discussed on the Event-Platform board.Mar 8 2023, 3:02 PM
JArguello-WMF moved this task from To be Estimated/To be discussed to Sprint 09 on the Event-Platform board.Mar 8 2023, 3:05 PM
JArguello-WMF edited projects, added Event-Platform (Sprint 09); removed Event-Platform.
JArguello-WMF edited projects, added Event-Platform (Sprint 10); removed Event-Platform (Sprint 09).Mar 13 2023, 2:17 PM
JArguello-WMF edited projects, added Event-Platform; removed Event-Platform (Sprint 10).Mar 13 2023, 3:41 PM
BTullis subscribed.Mar 15 2023, 9:52 AM
Ottomata moved this task from Backlog to To be Estimated/To be discussed on the Event-Platform board.Apr 19 2023, 3:05 PM
lbowmaker moved this task from To be Estimated/To be discussed to Sprint 12 on the Event-Platform board.Apr 19 2023, 3:16 PM
lbowmaker edited projects, added Event-Platform (Sprint 12); removed Event-Platform.
Ottomata updated the task description. (Show Details)May 3 2023, 2:29 PM
gmodena claimed this task.May 11 2023, 8:21 AM
gmodena moved this task from Next Up to In progress on the Event-Platform (Sprint 12) board.
elukey awarded a token.May 11 2023, 8:34 AM
gerritbot added a comment.May 11 2023, 3:12 PM

Change 919150 had a related patch set uploaded (by Gmodena; author: Gmodena):

[operations/deployment-charts@master] mediawiki-page-content-change-enrichment: use kafka TLS

https://gerrit.wikimedia.org/r/919150

gerritbot added a project: Patch-For-Review.May 11 2023, 3:12 PM
gmodena moved this task from In progress to In Review on the Event-Platform (Sprint 12) board.May 11 2023, 3:19 PM
gerritbot added a comment.May 11 2023, 4:46 PM

Change 919150 merged by jenkins-bot:

[operations/deployment-charts@master] mediawiki-page-content-change-enrichment: use kafka TLS

https://gerrit.wikimedia.org/r/919150

Maintenance_bot removed a project: Patch-For-Review.May 11 2023, 5:11 PM
gmodena updated the task description. (Show Details)May 11 2023, 6:46 PM
JArguello-WMF edited projects, added Event-Platform (Sprint 14 A); removed Event-Platform (Sprint 12).May 12 2023, 3:53 PM
JArguello-WMF moved this task from Next Up to In Review on the Event-Platform (Sprint 14 A) board.
JArguello-WMF moved this task from In Review to Done on the Event-Platform (Sprint 14 A) board.May 15 2023, 3:10 PM
gerritbot added a comment.May 15 2023, 7:59 PM

Change 919887 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/deployment-charts@master] dse-k8s-services/mediawiki-page-content-change-enrichment - use kafka SSL

https://gerrit.wikimedia.org/r/919887

gerritbot added a project: Patch-For-Review.May 15 2023, 7:59 PM

Change 919887 merged by Ottomata:

[operations/deployment-charts@master] dse-k8s-services/mediawiki-page-content-change-enrichment - use kafka SSL

https://gerrit.wikimedia.org/r/919887

Maintenance_bot removed a project: Patch-For-Review.May 15 2023, 8:10 PM
JArguello-WMF closed this task as Resolved.Jun 5 2023, 11:31 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL