Page MenuHomePhabricator
Create Task
Maniphest T326914

Migrate the WDQS streaming updater from FlinkKafkaConsumer/Producer to KafkaSource/Sink
Closed, ResolvedPublic
Actions

Description

FlinkKafkaConsumer and FlinkKafkaProducer are now deprecated, KafkaSource and KafkaSink should be used instead.
Unfortunately as these operators do store a state some migration steps have to be respected:

  • the job must be stopped with a savepoint and offsets committed to kafka
  • the job must be started from this savepoint with most probably --allowNonRestoredState set to true

Given the risk of failures it might be wise to do this in two steps:

  • add support for KafkaSink/KafkaSource in the codebase with an option to flip between the two implemations
  • drop usage of FlinkKafkaConsumer and FlinkKafkaProducer once the job is proven to run properly with KafkaSink/KafkaSource

AC:

  • FlinkKafkaConsumer and FlinkKafkaProducer usage are dropped from the codebase

Details

Show related patches Customize query in gerrit
TitleReferenceAuthorSource BranchDest Branch
Bump to 0.3.135repos/search-platform/flink-rdf-streaming-updater!12dcausse0.3.135main
Customize query in GitLab

Related Objects
Search...

Event Timeline

dcausse created this task.Jan 13 2023, 10:02 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 13 2023, 10:02 AM
Maintenance_bot added a project: Wikidata.Jan 13 2023, 10:29 AM
Gehel triaged this task as Medium priority.Jan 16 2023, 4:09 PM
Gehel moved this task from Incoming to Tech Debt on the Wikidata-Query-Service board.
dcausse mentioned this in T328925: Flink App Deployment monitoring.May 26 2023, 3:16 PM
dcausse added a project: Discovery-Search (Current work).Sep 14 2023, 2:15 PM
bking added a project: Data-Platform-SRE.Sep 14 2023, 2:28 PM
bking subscribed.
dcausse claimed this task.Sep 14 2023, 3:02 PM
dcausse moved this task from Incoming to In Progress on the Data-Platform-SRE board.
dcausse moved this task from Incoming to In Progress on the Discovery-Search (Current work) board.Sep 18 2023, 3:14 PM
gerritbot added a comment.Sep 18 2023, 5:46 PM

Change 958535 had a related patch set uploaded (by DCausse; author: DCausse):

[wikidata/query/rdf@master] Add an option to use newer KafkaSource/Sink flink APIs

https://gerrit.wikimedia.org/r/958535

gerritbot added a project: Patch-For-Review.Sep 18 2023, 5:46 PM
gerritbot added a comment.Sep 19 2023, 2:31 PM

Change 958947 had a related patch set uploaded (by DCausse; author: DCausse):

[wikidata/query/rdf@master] Drop deprecated FlinkKafka Consumer/Producer

https://gerrit.wikimedia.org/r/958947

dcausse moved this task from In Progress to Needs review on the Discovery-Search (Current work) board.Sep 19 2023, 2:56 PM
dr0ptp4kt subscribed.Sep 19 2023, 3:54 PM
gerritbot added a comment.Sep 20 2023, 1:35 PM

Change 959217 had a related patch set uploaded (by DCausse; author: DCausse):

[wikidata/query/deploy@master] flink-job: add support for a save and cancel action

https://gerrit.wikimedia.org/r/959217

gerritbot added a comment.Sep 20 2023, 1:35 PM

Change 959218 had a related patch set uploaded (by DCausse; author: DCausse):

[wikidata/query/deploy@master] flink: start using KafkaSource/Sink

https://gerrit.wikimedia.org/r/959218

gerritbot added a comment.Sep 21 2023, 10:15 AM

Change 959700 had a related patch set uploaded (by DCausse; author: DCausse):

[wikidata/query/rdf@master] Drop kafka offsets support in bootstrap and state extraction jobs

https://gerrit.wikimedia.org/r/959700

gerritbot added a comment.Sep 21 2023, 5:36 PM

Change 959700 merged by jenkins-bot:

[wikidata/query/rdf@master] Drop kafka offsets support in bootstrap and state extraction jobs

https://gerrit.wikimedia.org/r/959700

gerritbot added a comment.Sep 21 2023, 6:16 PM

Change 958535 merged by jenkins-bot:

[wikidata/query/rdf@master] Add an option to use newer KafkaSource/Sink flink APIs

https://gerrit.wikimedia.org/r/958535

gerritbot added a comment.Sep 26 2023, 2:39 PM

Change 961101 had a related patch set uploaded (by Bking; author: DCausse):

[wikidata/query/deploy@master] deploy version 0.3.130

https://gerrit.wikimedia.org/r/961101

gerritbot added a comment.Sep 26 2023, 2:42 PM

Change 959217 merged by DCausse:

[wikidata/query/deploy@master] flink-job: add support for a save and cancel action

https://gerrit.wikimedia.org/r/959217

gerritbot added a comment.Sep 26 2023, 2:42 PM

Change 961101 merged by DCausse:

[wikidata/query/deploy@master] deploy version 0.3.130

https://gerrit.wikimedia.org/r/961101

gerritbot added a comment.Sep 26 2023, 2:43 PM

Change 959218 merged by DCausse:

[wikidata/query/deploy@master] flink: start using KafkaSource/Sink

https://gerrit.wikimedia.org/r/959218

gerritbot added a comment.Sep 26 2023, 3:19 PM

Change 961137 had a related patch set uploaded (by DCausse; author: DCausse):

[wikidata/query/deploy@master] flink-job.py: add support for allowNonRestoredState

https://gerrit.wikimedia.org/r/961137

gerritbot added a comment.Sep 26 2023, 3:52 PM

Change 961144 had a related patch set uploaded (by DCausse; author: DCausse):

[wikidata/query/rdf@master] Kafka sources should rely on existing group offsets

https://gerrit.wikimedia.org/r/961144

gerritbot added a comment.Sep 26 2023, 4:55 PM

Change 961144 merged by jenkins-bot:

[wikidata/query/rdf@master] Kafka sources should rely on existing group offsets

https://gerrit.wikimedia.org/r/961144

dcausse moved this task from Needs review to To Be Deployed on the Discovery-Search (Current work) board.Sep 28 2023, 6:27 PM
gerritbot added a comment.Sep 29 2023, 3:12 PM

Change 962050 had a related patch set uploaded (by DCausse; author: DCausse):

[wikidata/query/rdf@master] Use different names for KafkaSink and FlinkKafkaProducer

https://gerrit.wikimedia.org/r/962050

gerritbot added a comment.Oct 2 2023, 6:54 PM

Change 962050 merged by jenkins-bot:

[wikidata/query/rdf@master] Use different names for KafkaSink and FlinkKafkaProducer

https://gerrit.wikimedia.org/r/962050

gerritbot added a comment.Oct 3 2023, 1:37 PM

Change 963057 had a related patch set uploaded (by DCausse; author: DCausse):

[operations/deployment-charts@master] rdf-streaming-updater: take a savepoint manually

https://gerrit.wikimedia.org/r/963057

gerritbot added a comment.Oct 3 2023, 1:41 PM

Change 963057 merged by jenkins-bot:

[operations/deployment-charts@master] rdf-streaming-updater: take a savepoint manually

https://gerrit.wikimedia.org/r/963057

gerritbot added a comment.Oct 3 2023, 4:46 PM

Change 963105 had a related patch set uploaded (by DCausse; author: DCausse):

[operations/deployment-charts@master] rdf-streaming-updater: simplify parallelism and use newer kafka APIs

https://gerrit.wikimedia.org/r/963105

gerritbot added a comment.Oct 3 2023, 4:58 PM

Change 963105 merged by Bking:

[operations/deployment-charts@master] rdf-streaming-updater: simplify parallelism and use newer kafka APIs

https://gerrit.wikimedia.org/r/963105

bking added a parent task: T342149: Test common operations in the flink operator/k8s/Flink ZK environment.Oct 3 2023, 5:18 PM
bking added a parent task: T340548: [EPIC] Deployment of the Search Update Pipeline on Flink / k8s.
bking mentioned this in T342149: Test common operations in the flink operator/k8s/Flink ZK environment.Oct 3 2023, 6:28 PM
dcausse added a comment.Oct 4 2023, 8:41 AM

After upgrading the test job to the new kafka connector APIs I now hit this problem:

  • org.apache.kafka.common.errors.UnknownProducerIdException: This exception is raised by the broker if it could not locate the producer metadata associated with the producerId in question. This could happen if, for instance, the producer's records were deleted because their retention time had elapsed. Once the last records of the producerId are removed, the producer's metadata is removed from the broker, and future appends by the producer will return this exception.

Consistently when doing the second checkpoints, the first checkpoint always succeed.

pfischer mentioned this in T348112: [Search Update Pipeline] pick kafka consumer offset defaults.Oct 4 2023, 9:15 AM
dcausse added a comment.Oct 4 2023, 1:25 PM

The new kafka connectors seem to do low level interactions with the kafka-clients (using java introspection to manipulate some states), looking further it appears that we force the use of kafka-clients 2.4.1 while the flink connector expects 3.2.3. Upgrading to this version and testing with yarn appears to fix the issue mentioned above.

gerritbot added a comment.Oct 4 2023, 2:02 PM

Change 963322 had a related patch set uploaded (by DCausse; author: DCausse):

[wikidata/query/rdf@master] Use kafka-clients 3.2.3 and set client.id.prefix

https://gerrit.wikimedia.org/r/963322

gerritbot added a comment.Oct 4 2023, 2:29 PM

Change 963322 merged by jenkins-bot:

[wikidata/query/rdf@master] Use kafka-clients 3.2.3 and set client.id.prefix

https://gerrit.wikimedia.org/r/963322

CodeReviewBot added a comment.Oct 4 2023, 3:01 PM

dcausse opened https://gitlab.wikimedia.org/repos/search-platform/flink-rdf-streaming-updater/-/merge_requests/12

Bump to 0.3.135

CodeReviewBot added a comment.Oct 4 2023, 3:07 PM

dcausse merged https://gitlab.wikimedia.org/repos/search-platform/flink-rdf-streaming-updater/-/merge_requests/12

Bump to 0.3.135

gerritbot added a comment.Oct 4 2023, 5:10 PM

Change 963383 had a related patch set uploaded (by DCausse; author: DCausse):

[operations/deployment-charts@master] rdf-streaming-updater: bump to 0.3.135

https://gerrit.wikimedia.org/r/963383

gerritbot added a comment.Oct 4 2023, 5:21 PM

Change 963383 merged by jenkins-bot:

[operations/deployment-charts@master] rdf-streaming-updater: bump to 0.3.135

https://gerrit.wikimedia.org/r/963383

gerritbot added a comment.Oct 5 2023, 5:36 PM

Change 961137 merged by DCausse:

[wikidata/query/deploy@master] flink-job.py: add support for allowNonRestoredState

https://gerrit.wikimedia.org/r/961137

dcausse moved this task from To Be Deployed to Needs Reporting on the Discovery-Search (Current work) board.Oct 10 2023, 9:52 AM

all jobs have been restarted to use newer kafka apis

Gehel closed this task as Resolved.Oct 13 2023, 12:58 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL