Page MenuHomePhabricator

Deploy Flink to kubernetes (k8s)
Open, HighPublic21 Estimated Story Points

Description

As on operator of WDQS / WCQS, I want to deploy Flink in a way that is robust so that we can deploy WDQS Streaming updater on top of it.

The strategy as discussed in part in T247058 is to use k8s to provide compute resources and Swift as storage. Communication with the teams in charge of k8s and Flink has already started, but our needs have to be formalized, and the concrete strategy for implementation needs to be defined.

AC:

  • Flink is deployed on a production k8s cluster
  • stream of TTL update is available in Kafka and ready to be consumed

Event Timeline

Gehel created this task.Sep 28 2020, 2:56 PM
Restricted Application added a project: Wikidata. · View Herald TranscriptSep 28 2020, 2:56 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Gehel updated the task description. (Show Details)Oct 5 2020, 3:31 PM
CBogen set the point value for this task to 21.Oct 5 2020, 5:09 PM
Mstyles added a subscriber: Mstyles.EditedOct 8 2020, 11:14 PM

We're having an in person meeting soon, but I just wanted to outline some things I think will need to happen from a high level. This is based on reading https://wikitech.wikimedia.org/wiki/Deployment_pipeline/Components.

  1. Create blubberfiles for the job manager
  2. Set up deployment pipeline with help from release engineering to accomodate for Java
  3. Create Helm charts

Other concerns include logging and monitoring

Gehel triaged this task as High priority.Oct 28 2020, 1:29 PM

After the helm chart is merged and published (both should happen automatically on a +2, I 've +1ed already), the final 2 items for deployment are:

Change 650309 had a related patch set uploaded (by Mstyles; owner: Mstyles):
[wikidata/query/flink-rdf-streaming-updater@master] add helm test and helm chart

https://gerrit.wikimedia.org/r/650309

Change 654723 had a related patch set uploaded (by Mstyles; owner: Mstyles):
[operations/deployment-charts@master] update flink logging

https://gerrit.wikimedia.org/r/654723