Page MenuHomePhabricator
Create Task
Maniphest T264006

Deploy Flink (rdf-streaming-updater) to kubernetes (k8s)
Closed, ResolvedPublic21 Estimated Story Points
Actions

Description

As on operator of WDQS / WCQS, I want to deploy Flink in a way that is robust so that we can deploy WDQS Streaming updater on top of it.

The strategy as discussed in part in T247058 is to use k8s to provide compute resources and Swift as storage. Communication with the teams in charge of k8s and Flink has already started, but our needs have to be formalized, and the concrete strategy for implementation needs to be defined.

The wikidata query service streaming updater is currently deployed to Yarn and is in beta. Current production plans are for the updater to be deployed in the Kubernetes cluster to the staging cluster and to the Eqiad cluster. No plans right now for multi-cluster deployment.

AC:

  • Flink is deployed on a production k8s cluster
  • stream of TTL update is available in Kafka and ready to be consumed

Details

Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

Gehel created this task.Sep 28 2020, 2:56 PM
Restricted Application added a project: Wikidata. · View Herald TranscriptSep 28 2020, 2:56 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Gehel updated the task description. (Show Details)Oct 5 2020, 3:31 PM
CBogen moved this task from Incoming to Current work on the Wikidata-Query-Service board.Oct 5 2020, 3:32 PM
CBogen added a project: Discovery-Search (Current work).
CBogen set the point value for this task to 21.Oct 5 2020, 5:09 PM
CBogen moved this task from Incoming to Ready for Dev -- SWE on the Discovery-Search (Current work) board.
Mstyles subscribed.EditedOct 8 2020, 11:14 PM

We're having an in person meeting soon, but I just wanted to outline some things I think will need to happen from a high level. This is based on reading https://wikitech.wikimedia.org/wiki/Deployment_pipeline/Components.

  1. Create blubberfiles for the job manager
  2. Set up deployment pipeline with help from release engineering to accomodate for Java
  3. Create Helm charts

Other concerns include logging and monitoring

Gehel triaged this task as High priority.Oct 28 2020, 1:29 PM
Mstyles moved this task from Ready for Dev -- SWE to Blocked/Waiting on the Discovery-Search (Current work) board.Oct 29 2020, 4:56 PM
Gehel added a subtask: T267310: Allow job name configuration for Streaming Updater.Nov 9 2020, 4:25 PM
Mstyles moved this task from Blocked/Waiting to Waiting on the Discovery-Search (Current work) board.Nov 30 2020, 6:09 PM
Gehel closed subtask T265512: Set up Pipeline Configuration in WDQS repo as Resolved.Dec 14 2020, 1:24 PM
Gehel closed subtask T265504: Create Blubberfile in WDQS repo as Resolved.Dec 14 2020, 2:03 PM
akosiaris subscribed.Dec 16 2020, 11:12 AM

After the helm chart is merged and published (both should happen automatically on a +2, I 've +1ed already), the final 2 items for deployment are:

gerritbot added a comment.Dec 18 2020, 12:15 AM

Change 650309 had a related patch set uploaded (by Mstyles; owner: Mstyles):
[wikidata/query/flink-rdf-streaming-updater@master] add helm test and helm chart

https://gerrit.wikimedia.org/r/650309

gerritbot added a project: Patch-For-Review.Dec 18 2020, 12:15 AM
gerritbot added a comment.Jan 7 2021, 1:19 AM

Change 654723 had a related patch set uploaded (by Mstyles; owner: Mstyles):
[operations/deployment-charts@master] update flink logging

https://gerrit.wikimedia.org/r/654723

gerritbot added a comment.Jan 21 2021, 10:20 AM

Change 654723 merged by jenkins-bot:
[operations/deployment-charts@master] update flink logging

https://gerrit.wikimedia.org/r/654723

Gehel closed subtask T269876: Add swift plugin to Flink k8s as Resolved.Feb 1 2021, 1:48 PM
Gehel closed subtask T265526: Create Helm Chart as Resolved.
Gehel added a subtask: T273735: Migrate Streaming Updater to newer Scala version.Feb 8 2021, 4:32 PM
Mstyles closed subtask T266495: Create Debian Package for Flink as Declined.Feb 8 2021, 10:32 PM
Gehel closed subtask T273097: Create Flink Base Image as Declined.Feb 10 2021, 10:02 AM
Gehel closed subtask T267310: Allow job name configuration for Streaming Updater as Resolved.Feb 17 2021, 3:55 PM
Gehel closed subtask T269619: Create pipelines for late/spurious/failed events as Resolved.Feb 24 2021, 1:27 PM
gerritbot added a comment.Feb 24 2021, 5:55 PM

Change 666713 had a related patch set uploaded (by Mstyles; owner: Mstyles):
[wikidata/query/rdf@master] update flink version to 1.12.1

https://gerrit.wikimedia.org/r/666713

gerritbot added a comment.Feb 25 2021, 8:01 AM

Change 666713 merged by jenkins-bot:
[wikidata/query/rdf@master] update flink version to 1.12.1

https://gerrit.wikimedia.org/r/666713

gerritbot added a comment.Mar 11 2021, 11:47 PM

Change 650309 merged by jenkins-bot:
[wikidata/query/flink-rdf-streaming-updater@master] add pipeline test and chart and update to Flink 1.12.1

https://gerrit.wikimedia.org/r/650309

gerritbot added a comment.Mar 12 2021, 6:02 PM

Change 671204 had a related patch set uploaded (by Mstyles; owner: Mstyles):
[operations/deployment-charts@master] create helmfile.d structure

https://gerrit.wikimedia.org/r/671204

Mstyles added a subtask: Restricted Task.Mar 12 2021, 7:47 PM
jeena closed subtask T277125: Allow timeout override for pipeline k8s deployments to ci cluster as Resolved.Mar 12 2021, 11:05 PM
Mstyles updated the task description. (Show Details)Mar 18 2021, 9:32 PM
Mstyles updated the task description. (Show Details)Mar 18 2021, 10:42 PM
Gehel closed subtask T276131: Fix Serialization Bug as Resolved.Mar 24 2021, 1:27 PM
Gehel closed subtask Restricted Task as Resolved.
Gehel mentioned this in T280382: WDQS hosts low on /srv disk space.Jun 7 2021, 1:09 PM
dcausse closed subtask T280166: Investigate using session cluster for Flink as Resolved.Jun 14 2021, 3:26 PM
Gehel closed subtask T278385: Streaming Updater must make all requests to proxy endpoints as Resolved.Jun 21 2021, 11:44 AM
jijiki updated the task description. (Show Details)Jun 22 2021, 8:56 AM
jijiki renamed this task from Deploy Flink to kubernetes (k8s) to Deploy Flink (rdf-streaming-updater) to kubernetes (k8s).Jun 22 2021, 9:30 AM
jijiki added a project: serviceops.
jijiki updated the task description. (Show Details)
jijiki added subscribers: • Zbyszko, dcausse.
Restricted Application added a project: [DEPRECATED] wdwb-tech. · View Herald TranscriptJun 22 2021, 9:30 AM
jijiki subscribed.Jun 22 2021, 9:32 AM

@Zbyszko, @dcausse, we will try to clear any leftovers this week so you can do a first deployment to staging. Since next week we will be switchover datacenrtres, we can't deploy flink to production clusters anhyway, until it is complete and every is ok

• Zbyszko added a comment.Jun 22 2021, 9:34 AM

@jijiki Great, thank you!

Ottomata subscribed.Jun 22 2021, 1:26 PM
Addshore moved this task from Inbox to External Realm on the [DEPRECATED] wdwb-tech board.Jun 28 2021, 2:54 PM
gerritbot added a comment.Jul 13 2021, 9:28 AM

Change 671204 merged by jenkins-bot:

[operations/deployment-charts@master] rdf-streaming-updater: switch to H/A session-cluster

https://gerrit.wikimedia.org/r/671204

gerritbot added a comment.Jul 14 2021, 1:03 PM

Change 704535 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):

[operations/deployment-charts@master] rdf-streaming-updater: add namespace for service

https://gerrit.wikimedia.org/r/704535

gerritbot added a comment.Jul 14 2021, 1:10 PM

Change 704326 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/deployment-charts@master] admin_ng: Add a new tiller ClusterRole for flink-session-cluster

https://gerrit.wikimedia.org/r/704326

gerritbot added a comment.Jul 14 2021, 1:23 PM

Change 704537 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):

[operations/puppet@production] Add rdf-streaming-updater user

https://gerrit.wikimedia.org/r/704537

gerritbot added a comment.Jul 14 2021, 1:31 PM

Change 704538 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):

[labs/private@master] Add tokens for rdf-streaming-updater service

https://gerrit.wikimedia.org/r/704538

gerritbot added a comment.Jul 14 2021, 1:37 PM

Change 704538 merged by Effie Mouzeli:

[labs/private@master] rdf-streaming-updater: Add tokens for service

https://gerrit.wikimedia.org/r/704538

jijiki mentioned this in rLPRI5ddf297a356c: rdf-streaming-updater: Add tokens for service.Jul 14 2021, 1:38 PM
gerritbot added a comment.Jul 14 2021, 1:51 PM

Change 704537 merged by Effie Mouzeli:

[operations/puppet@production] Add rdf-streaming-updater kubernetes user

https://gerrit.wikimedia.org/r/704537

gerritbot added a comment.Jul 14 2021, 2:05 PM

Change 704326 merged by jenkins-bot:

[operations/deployment-charts@master] admin_ng: Add a new tiller ClusterRole for flink-session-cluster

https://gerrit.wikimedia.org/r/704326

gerritbot added a comment.Jul 14 2021, 2:12 PM

Change 704535 merged by jenkins-bot:

[operations/deployment-charts@master] rdf-streaming-updater: add namespace for service

https://gerrit.wikimedia.org/r/704535

gerritbot added a comment.Jul 15 2021, 9:56 AM

Change 704757 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/deployment-charts@master] admin_ng: Fix name of rbac apiGroup in tiller-flink clusterrole

https://gerrit.wikimedia.org/r/704757

gerritbot added a comment.Jul 15 2021, 10:01 AM

Change 704757 merged by jenkins-bot:

[operations/deployment-charts@master] admin_ng: Fix name of rbac apiGroup in tiller-flink clusterrole

https://gerrit.wikimedia.org/r/704757

gerritbot added a comment.Jul 20 2021, 9:31 AM

Change 705633 had a related patch set uploaded (by DCausse; author: DCausse):

[wikidata/query/flink-rdf-streaming-updater@master] Add wmf-certificates

https://gerrit.wikimedia.org/r/705633

gerritbot added a comment.Jul 20 2021, 1:06 PM

Change 705671 had a related patch set uploaded (by DCausse; author: DCausse):

[operations/deployment-charts@master] rdf-streaming-updater: Do not use envoy for thanos-swift

https://gerrit.wikimedia.org/r/705671

gerritbot added a comment.Jul 20 2021, 2:29 PM

Change 705633 merged by jenkins-bot:

[wikidata/query/flink-rdf-streaming-updater@master] Switch to buster (java 11) and add wmf-certificates

https://gerrit.wikimedia.org/r/705633

gerritbot added a comment.Jul 20 2021, 2:54 PM

Change 705671 merged by jenkins-bot:

[operations/deployment-charts@master] rdf-streaming-updater: Do not use envoy for thanos-swift

https://gerrit.wikimedia.org/r/705671

gerritbot added a comment.Jul 20 2021, 4:31 PM

Change 705719 had a related patch set uploaded (by DCausse; author: DCausse):

[operations/deployment-charts@master] rdf-streaming-updater: use image version 2021-07-20-143040-production

https://gerrit.wikimedia.org/r/705719

dcausse moved this task from Waiting to In Progress on the Discovery-Search (Current work) board.Jul 20 2021, 4:34 PM
gerritbot added a comment.Jul 20 2021, 4:39 PM

Change 705719 merged by jenkins-bot:

[operations/deployment-charts@master] rdf-streaming-updater: use image version 2021-07-20-143040-production

https://gerrit.wikimedia.org/r/705719

gerritbot added a comment.Jul 26 2021, 12:04 PM

Change 708098 had a related patch set uploaded (by DCausse; author: DCausse):

[wikidata/query/flink-rdf-streaming-updater@master] properly install wmf-certificates on the prod image

https://gerrit.wikimedia.org/r/708098

gerritbot added a comment.Jul 26 2021, 12:12 PM

Change 708098 merged by jenkins-bot:

[wikidata/query/flink-rdf-streaming-updater@master] properly install wmf-certificates on the prod image

https://gerrit.wikimedia.org/r/708098

gerritbot added a comment.Jul 26 2021, 12:58 PM

Change 708111 had a related patch set uploaded (by DCausse; author: DCausse):

[operations/deployment-charts@master] rdf-streaming-updater: use image version 2021-07-26-125114-production

https://gerrit.wikimedia.org/r/708111

gerritbot added a comment.Jul 26 2021, 2:40 PM

Change 708111 merged by jenkins-bot:

[operations/deployment-charts@master] rdf-streaming-updater: use image version 2021-07-26-125114-production

https://gerrit.wikimedia.org/r/708111

gerritbot added a comment.Jul 28 2021, 1:46 PM

Change 708528 had a related patch set uploaded (by DCausse; author: DCausse):

[operations/deployment-charts@master] flink-session-cluster: fix main_app app label...

https://gerrit.wikimedia.org/r/708528

gerritbot added a comment.Jul 28 2021, 2:04 PM

Change 708528 merged by jenkins-bot:

[operations/deployment-charts@master] flink-session-cluster: fix main_app app label...

https://gerrit.wikimedia.org/r/708528

So9q subscribed.Aug 9 2021, 1:25 PM
So9q updated the task description. (Show Details)Aug 9 2021, 1:30 PM
dcausse updated the task description. (Show Details)Aug 23 2021, 3:59 PM
dcausse moved this task from In Progress to Needs Reporting on the Discovery-Search (Current work) board.
Gehel closed this task as Resolved.Aug 30 2021, 12:34 PM
Gehel claimed this task.
Gehel closed subtask T273098: High Availability Flink as Resolved.
Gehel closed subtask T273095: Deploy Helm Chart as Resolved.Sep 13 2021, 12:45 PM
Gehel closed subtask T280579: New service request: WDQS Flink based Streaming Updater as Resolved.Nov 8 2021, 2:56 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL