Page MenuHomePhabricator
Create Task
Maniphest T328675

Create a dse-k8s service demonstrating how to run the rdf-streaming-updater using the flink-app chart
Closed, ResolvedPublic5 Estimated Story Points
Actions

Description

The flink-k8s-operator is now available on the dse-k8s cluster. We should create a new helmfile using the flink-app chart to run an experimentation and learn how to operate a flink job using this technique.

AC:

  • a test job is running in the dse-k8s cluster

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
rdf-streaming-updater: tune mem overheadoperations/deployment-chartsmaster+2 -1
rdf-streaming-updater: bump image version to flink-1.16-rc3operations/deployment-chartsmaster+2 -2
Add debug code to identify classloading or serialization issueswikidata/query/rdfflink_1_16+5 -2
rdf-streaming-updater: bump job image to flink-1.16-rc2operations/deployment-chartsmaster+2 -3
flink-app: update to mesh.configuration 1.2.0operations/deployment-chartsmaster+3 -7
rdf-streaming-updater: temp fix, pin envoy image version to 1.18.3-2operations/deployment-chartsmaster+2 -0
[DNM] flink-app: always include /etc/envoy/ssl/ca.crtoperations/deployment-chartsmaster+4 -10
rdf-streaming-updater: use newer bootstrap stateoperations/deployment-chartsmaster+1 -1
flink-app: add envoy configmapsoperations/deployment-chartsmaster+3 -2
flink-app: include mesh.deployment.volumeoperations/deployment-chartsmaster+5 -4
rdf-streaming-updater: enable mesh sidecar containeroperations/deployment-chartsmaster+2 -0
rdf-streaming-updater: still use PLAINTEXT for kafkaoperations/deployment-chartsmaster+1 -1
rdf-streaming-updater: raise taskManager mem in dse-k8soperations/deployment-chartsmaster+1 -1
rdf-streaming-updater: use correct config path for dse-k8s-eqiadoperations/deployment-chartsmaster+1 -1
rdf-streaming-updater: use correct release and appoperations/deployment-chartsmaster+3 -3
rdf-streaming-updater: fix docker image URLoperations/deployment-chartsmaster+1 -1
rdf-streaming-updater: jarURI should use local:// not file://operations/deployment-chartsmaster+1 -1
rdf-streaming-updater: add jarURIoperations/deployment-chartsmaster+1 -0
dse-k8s-eqiad: flink-operator should watch rdf-streaming-updateroperations/deployment-chartsmaster+2 -0
rdf-streaming-update: use correct data typeoperations/deployment-chartsmaster+3 -3
rdf-streaming-updater: use correct resource nameoperations/deployment-chartsmaster+1 -1
rdf-streaming-updater: add a test job using the k8s operator...operations/deployment-chartsmaster+194 -0
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
dcausse mentioned this in T326409: Migrate the wdqs streaming updater flink jobs to flink-k8s-operator deployment model.Feb 2 2023, 3:04 PM
gerritbot added a comment.Feb 2 2023, 4:36 PM

Change 886005 had a related patch set uploaded (by DCausse; author: DCausse):

[operations/deployment-charts@master] [WIP] rdf-streaming-updater: add a test job using the k8s operator...

https://gerrit.wikimedia.org/r/886005

gerritbot added a project: Patch-For-Review.Feb 2 2023, 4:36 PM
Maintenance_bot added a project: Wikidata.Feb 2 2023, 4:45 PM
pfischer subscribed.Feb 13 2023, 4:30 PM
Gehel moved this task from Incoming to Current work on the Wikidata-Query-Service board.Feb 13 2023, 4:33 PM
Gehel added a project: Discovery-Search (Current work).
dcausse updated the task description. (Show Details)Feb 13 2023, 4:48 PM
Gehel updated the task description. (Show Details)Feb 13 2023, 4:49 PM
Gehel moved this task from Incoming to Ready for Dev -- SRE/Ops on the Discovery-Search (Current work) board.
Gehel set the point value for this task to 5.
dcausse claimed this task.Feb 14 2023, 9:03 AM
dcausse moved this task from incoming to in progress on the Wikidata board.
dcausse moved this task from in progress to incoming on the Wikidata board.Feb 14 2023, 9:06 AM
dcausse moved this task from Ready for Dev -- SRE/Ops to In Progress on the Discovery-Search (Current work) board.
gerritbot added a comment.Mar 9 2023, 3:34 PM

Change 886005 merged by jenkins-bot:

[operations/deployment-charts@master] rdf-streaming-updater: add a test job using the k8s operator...

https://gerrit.wikimedia.org/r/886005

Maintenance_bot removed a project: Patch-For-Review.Mar 9 2023, 4:10 PM
gerritbot added a comment.Mar 20 2023, 2:23 PM

Change 901218 had a related patch set uploaded (by Bking; author: Bking):

[operations/deployment-charts@master] rdf-streaming-updater: use correct resource name

https://gerrit.wikimedia.org/r/901218

gerritbot added a project: Patch-For-Review.Mar 20 2023, 2:23 PM
gerritbot added a comment.Mar 20 2023, 2:34 PM

Change 901218 merged by jenkins-bot:

[operations/deployment-charts@master] rdf-streaming-updater: use correct resource name

https://gerrit.wikimedia.org/r/901218

gerritbot added a comment.Mar 20 2023, 2:42 PM

Change 901226 had a related patch set uploaded (by Bking; author: Bking):

[operations/deployment-charts@master] rdf-streaming-update: use correct data type

https://gerrit.wikimedia.org/r/901226

gerritbot added a comment.Mar 20 2023, 2:49 PM

Change 901226 merged by Bking:

[operations/deployment-charts@master] rdf-streaming-update: use correct data type

https://gerrit.wikimedia.org/r/901226

Maintenance_bot removed a project: Patch-For-Review.Mar 20 2023, 3:10 PM
gerritbot added a comment.Mar 20 2023, 3:43 PM

Change 901240 had a related patch set uploaded (by Bking; author: Bking):

[operations/deployment-charts@master] rdf-streaming-updater: use correct release and app

https://gerrit.wikimedia.org/r/901240

gerritbot added a project: Patch-For-Review.Mar 20 2023, 3:43 PM
Gehel merged a task: T321491: Evaluate Flink Operator on DSE Kubernetes Cluster for deployment and management of stateful search applications.Mar 20 2023, 4:13 PM
Gehel added subscribers: Ottomata, BTullis, JMeybohm, gmodena.
gerritbot added a comment.Mar 20 2023, 5:38 PM

Change 901253 had a related patch set uploaded (by DCausse; author: DCausse):

[operations/deployment-charts@master] dse-k8s-eqiad: flink-operator should watch rdf-streaming-updater

https://gerrit.wikimedia.org/r/901253

gerritbot added a comment.Mar 20 2023, 6:53 PM

Change 901253 merged by jenkins-bot:

[operations/deployment-charts@master] dse-k8s-eqiad: flink-operator should watch rdf-streaming-updater

https://gerrit.wikimedia.org/r/901253

gerritbot added a comment.Mar 21 2023, 2:54 PM

Change 901608 had a related patch set uploaded (by DCausse; author: DCausse):

[operations/deployment-charts@master] rdf-streaming-updater: add jarURI

https://gerrit.wikimedia.org/r/901608

gerritbot added a comment.Mar 21 2023, 3:11 PM

Change 901608 merged by jenkins-bot:

[operations/deployment-charts@master] rdf-streaming-updater: add jarURI

https://gerrit.wikimedia.org/r/901608

gerritbot added a comment.Mar 21 2023, 3:25 PM

Change 901617 had a related patch set uploaded (by DCausse; author: DCausse):

[operations/deployment-charts@master] rdf-streaming-updater: jarURI should use local:// not file://

https://gerrit.wikimedia.org/r/901617

gerritbot added a comment.Mar 21 2023, 3:33 PM

Change 901617 merged by jenkins-bot:

[operations/deployment-charts@master] rdf-streaming-updater: jarURI should use local:// not file://

https://gerrit.wikimedia.org/r/901617

gerritbot added a comment.Mar 21 2023, 3:39 PM

Change 901619 had a related patch set uploaded (by DCausse; author: DCausse):

[operations/deployment-charts@master] rdf-streaming-updater: fix docker image URL

https://gerrit.wikimedia.org/r/901619

gerritbot added a comment.Mar 21 2023, 3:46 PM

Change 901619 merged by jenkins-bot:

[operations/deployment-charts@master] rdf-streaming-updater: fix docker image URL

https://gerrit.wikimedia.org/r/901619

gerritbot added a comment.Mar 22 2023, 2:16 PM

Change 901240 merged by jenkins-bot:

[operations/deployment-charts@master] rdf-streaming-updater: use correct release and app

https://gerrit.wikimedia.org/r/901240

Maintenance_bot removed a project: Patch-For-Review.Mar 22 2023, 2:30 PM
gerritbot added a comment.Mar 27 2023, 2:29 PM

Change 903266 had a related patch set uploaded (by Bking; author: Bking):

[operations/deployment-charts@master] rdf-streaming-updater: use correct config path for dse-k8s-eqiad

https://gerrit.wikimedia.org/r/903266

gerritbot added a project: Patch-For-Review.Mar 27 2023, 2:29 PM
gerritbot added a comment.Mar 27 2023, 2:35 PM

Change 903266 merged by jenkins-bot:

[operations/deployment-charts@master] rdf-streaming-updater: use correct config path for dse-k8s-eqiad

https://gerrit.wikimedia.org/r/903266

Maintenance_bot removed a project: Patch-For-Review.Mar 27 2023, 3:10 PM
gerritbot added a comment.Mar 28 2023, 2:46 PM

Change 903659 had a related patch set uploaded (by Bking; author: Bking):

[operations/deployment-charts@master] rdf-streaming-updater: raise taskManager mem in dse-k8s

https://gerrit.wikimedia.org/r/903659

gerritbot added a project: Patch-For-Review.Mar 28 2023, 2:46 PM
gerritbot added a comment.Mar 28 2023, 2:54 PM

Change 903659 merged by jenkins-bot:

[operations/deployment-charts@master] rdf-streaming-updater: raise taskManager mem in dse-k8s

https://gerrit.wikimedia.org/r/903659

Maintenance_bot removed a project: Patch-For-Review.Mar 28 2023, 3:11 PM
gerritbot added a comment.Mar 28 2023, 3:40 PM

Change 903689 had a related patch set uploaded (by DCausse; author: DCausse):

[operations/deployment-charts@master] rdf-streaming-updater: still use PLAINTEXT for kafka

https://gerrit.wikimedia.org/r/903689

gerritbot added a project: Patch-For-Review.Mar 28 2023, 3:40 PM
gerritbot added a comment.Mar 28 2023, 3:48 PM

Change 903689 merged by jenkins-bot:

[operations/deployment-charts@master] rdf-streaming-updater: still use PLAINTEXT for kafka

https://gerrit.wikimedia.org/r/903689

Maintenance_bot removed a project: Patch-For-Review.Mar 28 2023, 4:10 PM
gerritbot added a comment.Mar 28 2023, 4:13 PM

Change 903696 had a related patch set uploaded (by DCausse; author: DCausse):

[operations/deployment-charts@master] rdf-streaming-updater: enable mesh sidecar container

https://gerrit.wikimedia.org/r/903696

gerritbot added a project: Patch-For-Review.Mar 28 2023, 4:13 PM
gerritbot added a comment.Mar 28 2023, 4:59 PM

Change 903696 merged by jenkins-bot:

[operations/deployment-charts@master] rdf-streaming-updater: enable mesh sidecar container

https://gerrit.wikimedia.org/r/903696

Maintenance_bot removed a project: Patch-For-Review.Mar 28 2023, 5:10 PM
gerritbot added a comment.Mar 28 2023, 5:29 PM

Change 903729 had a related patch set uploaded (by DCausse; author: DCausse):

[operations/deployment-charts@master] flink-app: include mesh.deployment.volume

https://gerrit.wikimedia.org/r/903729

gerritbot added a project: Patch-For-Review.Mar 28 2023, 5:29 PM
gerritbot added a comment.Mar 28 2023, 5:53 PM

Change 903729 merged by jenkins-bot:

[operations/deployment-charts@master] flink-app: include mesh.deployment.volume

https://gerrit.wikimedia.org/r/903729

Maintenance_bot removed a project: Patch-For-Review.Mar 28 2023, 6:10 PM
gerritbot added a comment.Mar 28 2023, 6:14 PM

Change 903740 had a related patch set uploaded (by DCausse; author: DCausse):

[operations/deployment-charts@master] flink-app: add envoy configmaps

https://gerrit.wikimedia.org/r/903740

gerritbot added a project: Patch-For-Review.Mar 28 2023, 6:14 PM
gerritbot added a comment.Mar 29 2023, 2:05 PM

Change 904188 had a related patch set uploaded (by DCausse; author: DCausse):

[operations/deployment-charts@master] rdf-streaming-updater: use newer bootstrap state

https://gerrit.wikimedia.org/r/904188

gerritbot added a comment.Mar 29 2023, 2:10 PM

Change 903740 merged by jenkins-bot:

[operations/deployment-charts@master] flink-app: add envoy configmaps

https://gerrit.wikimedia.org/r/903740

gerritbot added a comment.Mar 29 2023, 2:12 PM

Change 904188 merged by jenkins-bot:

[operations/deployment-charts@master] rdf-streaming-updater: use newer bootstrap state

https://gerrit.wikimedia.org/r/904188

Maintenance_bot removed a project: Patch-For-Review.Mar 29 2023, 2:30 PM
gerritbot added a comment.Mar 30 2023, 9:12 AM

Change 904464 had a related patch set uploaded (by DCausse; author: DCausse):

[operations/deployment-charts@master] [DNM] flink-app: always include /etc/envoy/ssl/ca.crt

https://gerrit.wikimedia.org/r/904464

gerritbot added a project: Patch-For-Review.Mar 30 2023, 9:12 AM
gerritbot added a comment.Mar 30 2023, 12:27 PM

Change 904464 abandoned by DCausse:

[operations/deployment-charts@master] [DNM] flink-app: always include /etc/envoy/ssl/ca.crt

Reason:

Installing the wmf-certificates package is better solution https://phabricator.wikimedia.org/T333551

https://gerrit.wikimedia.org/r/904464

Maintenance_bot removed a project: Patch-For-Review.Mar 30 2023, 12:30 PM
gerritbot added a comment.Mar 30 2023, 2:49 PM

Change 904553 had a related patch set uploaded (by DCausse; author: DCausse):

[operations/deployment-charts@master] rdf-streaming-updater: temp fix, pin envoy image version to 1.18.3-2

https://gerrit.wikimedia.org/r/904553

gerritbot added a project: Patch-For-Review.Mar 30 2023, 2:49 PM
gerritbot added a comment.Mar 30 2023, 2:55 PM

Change 904553 merged by jenkins-bot:

[operations/deployment-charts@master] rdf-streaming-updater: temp fix, pin envoy image version to 1.18.3-2

https://gerrit.wikimedia.org/r/904553

Maintenance_bot removed a project: Patch-For-Review.Mar 30 2023, 3:10 PM
gerritbot added a comment.Mar 30 2023, 4:37 PM

dcausse opened https://gitlab.wikimedia.org/repos/search-platform/flink-rdf-streaming-updater/-/merge_requests/3

Do not store the job jar in usrlib

gerritbot added a project: Patch-For-Review.Mar 30 2023, 4:37 PM
gerritbot added a comment.Mar 31 2023, 9:15 AM

dcausse merged https://gitlab.wikimedia.org/repos/search-platform/flink-rdf-streaming-updater/-/merge_requests/3

Do not store the job jar in usrlib

Maintenance_bot removed a project: Patch-For-Review.Mar 31 2023, 9:30 AM
gerritbot added a comment.Mar 31 2023, 10:02 AM

Change 904762 had a related patch set uploaded (by DCausse; author: DCausse):

[operations/deployment-charts@master] flink-app: update to mesh.configuration 1.2.0

https://gerrit.wikimedia.org/r/904762

gerritbot added a project: Patch-For-Review.Mar 31 2023, 10:02 AM

Change 904763 had a related patch set uploaded (by DCausse; author: DCausse):

[operations/deployment-charts@master] rdf-streaming-updater: bump job image to flink-1.16-rc2...

https://gerrit.wikimedia.org/r/904763

gerritbot added a comment.Mar 31 2023, 12:50 PM

Change 904762 merged by jenkins-bot:

[operations/deployment-charts@master] flink-app: update to mesh.configuration 1.2.0

https://gerrit.wikimedia.org/r/904762

gerritbot added a comment.Mar 31 2023, 1:02 PM

Change 904763 merged by jenkins-bot:

[operations/deployment-charts@master] rdf-streaming-updater: bump job image to flink-1.16-rc2

https://gerrit.wikimedia.org/r/904763

Maintenance_bot removed a project: Patch-For-Review.Mar 31 2023, 1:10 PM
dcausse added a comment.Mar 31 2023, 1:20 PM

currently still stuck with

scala.MatchError: None (of class scala.None$)
	at org.wikidata.query.rdf.updater.SideOutputSerializationSchema.getRecordClock(SideOutputSerializationSchema.scala:31)
	at org.wikidata.query.rdf.updater.SideOutputSerializationSchema.clock$lzycompute(SideOutputSerializationSchema.scala:40)
	at org.wikidata.query.rdf.updater.SideOutputSerializationSchema.clock(SideOutputSerializationSchema.scala:40)
	at org.wikidata.query.rdf.updater.SideOutputSerializationSchema.serialize(SideOutputSerializationSchema.scala:73)
	at org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer.invoke(FlinkKafkaProducer.java:918)
	at org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer.invoke(FlinkKafkaProducer.java:101)
	at org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction.invoke(TwoPhaseCommitSinkFunction.java:245)
	at org.apache.flink.streaming.api.operators.StreamSink.processElement(StreamSink.java:54)
	at org.apache.flink.streaming.runtime.tasks.ChainingOutput.pushToOperator(ChainingOutput.java:99)
	at org.apache.flink.streaming.runtime.tasks.ChainingOutput.collect(ChainingOutput.java:86)
	at org.apache.flink.streaming.runtime.tasks.CopyingBroadcastingOutputCollector.collect(CopyingBroadcastingOutputCollector.java:60)
	at org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:62)
	at org.apache.flink.streaming.api.operators.KeyedProcessOperator$ContextImpl.output(KeyedProcessOperator.java:129)
	at org.wikidata.query.rdf.updater.ReorderAndDecideMutationOperation.fireEvent(ReorderAndDecideMutationOperation.scala:112)
	at org.wikidata.query.rdf.updater.ReorderAndDecideMutationOperation.processElement(ReorderAndDecideMutationOperation.scala:29)
	at org.wikidata.query.rdf.updater.ReorderAndDecideMutationOperation.processElement(ReorderAndDecideMutationOperation.scala:12)
	at org.apache.flink.streaming.api.operators.KeyedProcessOperator.processElement(KeyedProcessOperator.java:83)
	at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask$StreamTaskNetworkOutput.emitRecord(OneInputStreamTask.java:233)
	at org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput.processElement(AbstractStreamTaskNetworkInput.java:134)
	at org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput.emitNext(AbstractStreamTaskNetworkInput.java:105)
	at org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:65)
	at org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:542)
	at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:231)
	at org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:831)
	at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:780)
	at org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:935)
	at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:914)
	at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:728)
	at org.apache.flink.runtime.taskmanager.Task.run(Task.java:550)
	at java.base/java.lang.Thread.run(Thread.java:829)

suspecting a classloader problem...

gerritbot added a comment.Mar 31 2023, 1:47 PM

Change 904797 had a related patch set uploaded (by DCausse; author: DCausse):

[wikidata/query/rdf@flink_1_16] Add debug code to identify classloading or serialization issues

https://gerrit.wikimedia.org/r/904797

gerritbot added a project: Patch-For-Review.Mar 31 2023, 1:47 PM
gerritbot added a comment.Mar 31 2023, 2:42 PM

dcausse opened https://gitlab.wikimedia.org/repos/search-platform/flink-rdf-streaming-updater/-/merge_requests/4

temp hack to investigate classpath issue

gerritbot added a comment.Mar 31 2023, 2:43 PM

Change 904797 merged by jenkins-bot:

[wikidata/query/rdf@flink_1_16] Add debug code to identify classloading or serialization issues

https://gerrit.wikimedia.org/r/904797

gerritbot added a comment.Mar 31 2023, 2:44 PM

dcausse merged https://gitlab.wikimedia.org/repos/search-platform/flink-rdf-streaming-updater/-/merge_requests/4

temp hack to investigate classpath issue

gerritbot added a comment.Mar 31 2023, 2:52 PM

Change 904813 had a related patch set uploaded (by DCausse; author: DCausse):

[operations/deployment-charts@master] rdf-streaming-updater: bump image version to flink-1.16-rc3

https://gerrit.wikimedia.org/r/904813

gerritbot added a comment.Apr 3 2023, 12:39 PM

Change 904813 merged by jenkins-bot:

[operations/deployment-charts@master] rdf-streaming-updater: bump image version to flink-1.16-rc3

https://gerrit.wikimedia.org/r/904813

gerritbot added a comment.Apr 4 2023, 11:53 AM

Change 905602 had a related patch set uploaded (by DCausse; author: DCausse):

[operations/deployment-charts@master] rdf-streaming-updater: tune mem overhead

https://gerrit.wikimedia.org/r/905602

gerritbot added a comment.Apr 4 2023, 2:33 PM

Change 905602 merged by jenkins-bot:

[operations/deployment-charts@master] rdf-streaming-updater: tune mem overhead

https://gerrit.wikimedia.org/r/905602

Maintenance_bot removed a project: Patch-For-Review.Apr 4 2023, 3:10 PM
dcausse moved this task from In Progress to Needs Reporting on the Discovery-Search (Current work) board.Apr 4 2023, 6:18 PM

the flink job is properly running from the k8s-dse cluster using the flink-operator: https://grafana.wikimedia.org/d/gCFgfpG7k/flink-cluster?orgId=1&var-datasource=eqiad+prometheus%2Fk8s-dse&var-namespace=rdf-streaming-updater&from=now-3h&to=now.

Gehel closed this task as Resolved.Apr 7 2023, 9:40 AM
bking added a subtask: T334244: Upgrade the wdqs updater jobs to flink 1.16.1.Apr 10 2023, 3:43 PM
Gehel closed subtask T334244: Upgrade the wdqs updater jobs to flink 1.16.1 as Resolved.Jun 2 2023, 9:51 AM
bking mentioned this in T344614: Add Zookeeper config to 'rdf-streaming-updater' test service on DSE cluster.Aug 21 2023, 2:32 PM
bking mentioned this in T317045: [Epic] Re-architect the Search Update Pipeline.Sep 6 2023, 2:17 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits