Page MenuHomePhabricator
Create Task
Maniphest T333373

The WDQS streaming updater should support connecting to kafka with SSL
Closed, ResolvedPublic
Actions

Description

The way we configure the kafka consumer assume that we pass the PLAINTEXT port.
We should have a way to configure the security protocol to use when connecting to a kafka broker.

Multiple options:

  • infer this from the port, 9092->PLAINTEXT, 9093 -> SSL
  • accept URI like brokers: ssl://IP:PORT, plaintext://IP:PORT
  • separate config option, broker_security_protocol

AC:

  • the streaming updater is able to connect to kafka using the SSL protocol

Details

SubjectRepoBranchLines +/-
rdf-streaming-updater: use SSL to connect to kafka-mainoperations/deployment-chartsmaster+6 -6
Properly pass kafka client options to all producerswikidata/query/rdfmaster+6 -4
rdf-streaming-updater: revert to using plaintext with kafka-mainoperations/deployment-chartsmaster+5 -5
rdf-streaming-updater: use SSL and external-services fqdn to access kafka-mainoperations/deployment-chartsmaster+8 -6
Customize query in gerrit

Event Timeline

• dcausse created this task.Mar 28 2023, 3:33 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 28 2023, 3:33 PM
• dcausse updated the task description. (Show Details)Mar 28 2023, 3:37 PM
Maintenance_bot added a project: Wikidata.Mar 28 2023, 3:45 PM
Gehel triaged this task as Medium priority.Apr 3 2023, 3:41 PM
Gehel moved this task from Incoming to Tech Debt on the Wikidata-Query-Service board.
Lydia_Pintscher moved this task from incoming to monitoring on the Wikidata board.Apr 6 2023, 10:10 AM
• dcausse claimed this task.Sep 23 2024, 12:42 PM
• dcausse moved this task from Tech Debt to Current work on the Wikidata-Query-Service board.
• dcausse added a project: Discovery-Search (Current work).
• dcausse moved this task from Incoming to In Progress on the Discovery-Search (Current work) board.
CodeReviewBot added a project: Patch-For-Review.Sep 23 2024, 2:03 PM

dcausse opened https://gitlab.wikimedia.org/repos/search-platform/flink-rdf-streaming-updater/-/merge_requests/18

Bump to 0.3.147

CodeReviewBot added a comment.Sep 23 2024, 2:32 PM

dcausse merged https://gitlab.wikimedia.org/repos/search-platform/flink-rdf-streaming-updater/-/merge_requests/18

Bump to 0.3.147

Maintenance_bot removed a project: Patch-For-Review.Sep 23 2024, 3:32 PM
gerritbot added a comment.Sep 23 2024, 4:20 PM

Change #1072231 had a related patch set uploaded (by DCausse; author: DCausse):

[operations/deployment-charts@master] rdf-streaming-updater: use SSL and external-services fqdn to access kafka-main

https://gerrit.wikimedia.org/r/1072231

gerritbot added a project: Patch-For-Review.Sep 23 2024, 4:20 PM
• dcausse moved this task from In Progress to To Be Deployed on the Discovery-Search (Current work) board.Sep 23 2024, 4:21 PM
gerritbot added a comment.Sep 25 2024, 7:46 AM

Change #1072231 merged by jenkins-bot:

[operations/deployment-charts@master] rdf-streaming-updater: use SSL and external-services fqdn to access kafka-main

https://gerrit.wikimedia.org/r/1072231

• dcausse moved this task from To Be Deployed to Needs Reporting on the Discovery-Search (Current work) board.Sep 25 2024, 8:07 AM
Maintenance_bot removed a project: Patch-For-Review.Sep 25 2024, 8:32 AM
• dcausse moved this task from Needs Reporting to In Progress on the Discovery-Search (Current work) board.Sep 25 2024, 1:09 PM
gerritbot added a comment.Sep 25 2024, 1:09 PM

Change #1075561 had a related patch set uploaded (by DCausse; author: DCausse):

[operations/deployment-charts@master] rdf-streaming-updater: revert to using plaintext with kafka-main

https://gerrit.wikimedia.org/r/1075561

gerritbot added a project: Patch-For-Review.Sep 25 2024, 1:09 PM
gerritbot added a comment.Sep 25 2024, 1:12 PM

Change #1075562 had a related patch set uploaded (by DCausse; author: DCausse):

[wikidata/query/rdf@master] Properly pass kafka client options to all producers

https://gerrit.wikimedia.org/r/1075562

gerritbot added a comment.Sep 25 2024, 1:18 PM

Change #1075561 merged by jenkins-bot:

[operations/deployment-charts@master] rdf-streaming-updater: revert to using plaintext with kafka-main

https://gerrit.wikimedia.org/r/1075561

• dcausse moved this task from In Progress to Needs review on the Discovery-Search (Current work) board.Oct 1 2024, 12:30 PM
gerritbot added a comment.Oct 1 2024, 3:22 PM

Change #1075562 merged by jenkins-bot:

[wikidata/query/rdf@master] Properly pass kafka client options to all producers

https://gerrit.wikimedia.org/r/1075562

Maintenance_bot removed a project: Patch-For-Review.Oct 1 2024, 3:31 PM
• dcausse moved this task from Needs review to To Be Deployed on the Discovery-Search (Current work) board.Oct 2 2024, 1:11 PM
CodeReviewBot added a project: Patch-For-Review.Oct 2 2024, 1:12 PM

dcausse opened https://gitlab.wikimedia.org/repos/search-platform/flink-rdf-streaming-updater/-/merge_requests/19

Bump to 0.3.148

CodeReviewBot added a comment.Oct 2 2024, 1:14 PM

dcausse merged https://gitlab.wikimedia.org/repos/search-platform/flink-rdf-streaming-updater/-/merge_requests/19

Bump to 0.3.148

Maintenance_bot removed a project: Patch-For-Review.Oct 2 2024, 1:32 PM
gerritbot added a comment.Oct 2 2024, 1:56 PM

Change #1077396 had a related patch set uploaded (by DCausse; author: DCausse):

[operations/deployment-charts@master] rdf-streaming-updater: use SSL to connect to kafka-main

https://gerrit.wikimedia.org/r/1077396

gerritbot added a project: Patch-For-Review.Oct 2 2024, 1:56 PM
gerritbot added a comment.Oct 3 2024, 9:32 AM

Change #1077396 merged by jenkins-bot:

[operations/deployment-charts@master] rdf-streaming-updater: use SSL to connect to kafka-main

https://gerrit.wikimedia.org/r/1077396

• dcausse moved this task from To Be Deployed to Needs Reporting on the Discovery-Search (Current work) board.Oct 3 2024, 9:47 AM
Maintenance_bot removed a project: Patch-For-Review.Oct 3 2024, 10:30 AM
Gehel closed this task as Resolved.Oct 4 2024, 7:45 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits