Page MenuHomePhabricator
Create Task
Maniphest T326409

Migrate the wdqs streaming updater flink jobs to flink-k8s-operator deployment model
Closed, ResolvedPublic0 Estimated Story Points
Actions

Description

The WDQS streaming updater is using a session cluster mode but the work done on T324576 should allow us to use a better approach based on the flink-k8s-operator.

Once the operator available we should migrate the WDQS streaming updater to this new deployment model.

Preparation steps:

  • T304914 fully remove any references to swift:// resources in eqiad
  • T289836 deploy flink 1.16 to the existing flink-session-cluster deployments in wikikube and run the streaming updater on top of it
  • Test the updater job on the dse-k8s cluster
    • create a namespace for the rdf-streaming-updater on the dse-k8s cluster: https://gerrit.wikimedia.org/r/c/operations/puppet/+/882748
    • T328675 create a helmfile service using the FlinkDeployment resource via the flink-app helm chart
    • T341792 Provision Zookeeper Cluster for storing Flink HA data
    • T344614 Add Zookeeper config to 'rdf-streaming-updater' test service on DSE cluster
    • test various maintenance operations: taking savepoint, job upgrade, H/A recoveries (kill pods manually), k8s upgrade (wipe out the namespace, T293063), ... (see also T342149, T328561)
  • Enable the k8s-operator on the staging wikikube cluster for the rdf-streaming-updater namespace (might need a dedicated task)
    • test various maintenance operations on staging wk: taking savepoint, job upgrade, H/A recoveries (kill pods manually), k8s upgrade (wipe out the namespace, T293063), ... (see also T328561)
  • Enable the k8s-operator on the production wikikube cluster for the rdf-streaming-updater namespace (might need a dedicated task)

Migration steps:

  • eqiad:
    • depool WDQS&WCQS read traffic in eqiad
    • stop the production jobs with a savepoint using the python tools
    • update the k8s chart to set the right savepoint path for both jobs
    • deploy the jobs
    • repool WDQS&WCQS in eqiad
  • codfw:
    • depool WDQS&WCQS read traffic in codfw
    • stop the production jobs with a savepoint using the python tools
    • update the k8s chart to set the right savepoint path for both jobs
    • deploy the jobs
    • repool WDQS&WCQS in codfw

Cleanups: moved to T350784

AC:

Details

SubjectRepoBranchLines +/-
Revert "Revert "Remove flink RBAC snowflakes""operations/deployment-chartsmaster+0 -34
Revert "Remove flink RBAC snowflakes"operations/deployment-chartsmaster+34 -0
Remove flink RBAC snowflakesoperations/deployment-chartsmaster+0 -34
dse-k8s: add rdf-streaming-updater namespaceoperations/puppetproduction+4 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
dcausse updated the task description. (Show Details)Jan 11 2023, 3:50 PM
gerritbot added a comment.Jan 19 2023, 3:52 PM

dcausse opened https://gitlab.wikimedia.org/repos/search-platform/flink-rdf-streaming-updater/-/merge_requests/1

Initial commit

Ottomata subscribed.Jan 20 2023, 12:16 PM

What namespace strategy should we use for flink jobs? A single one for all wmf flink jobs, per team, per project?

One per application is what I'm expecting.

Ottomata added a subscriber: gmodena.Jan 20 2023, 12:16 PM
JMeybohm subscribed.Jan 23 2023, 12:24 PM
bking added a subtask: T324576: Flink on Kubernetes Helm charts.Jan 23 2023, 8:07 PM
dcausse updated the task description. (Show Details)Jan 24 2023, 2:04 PM
dcausse updated the task description. (Show Details)Jan 24 2023, 2:14 PM
gerritbot added a comment.Jan 26 2023, 2:46 PM

Change 882748 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] dse-k8s: add rdf-streaming-updater namespace

https://gerrit.wikimedia.org/r/882748

gerritbot added a project: Patch-For-Review.Jan 26 2023, 2:46 PM

Change 882748 merged by Bking:

[operations/puppet@production] dse-k8s: add rdf-streaming-updater namespace

https://gerrit.wikimedia.org/r/882748

bking updated the task description. (Show Details)Jan 26 2023, 2:48 PM
Ottomata added a subscriber: BTullis.Jan 26 2023, 3:03 PM

create a namespace for the rdf-streaming-updater on the dse-k8s cluster

BTW, I _think_ there is more involved than just this Puppet patch? Reach out to @BTullis ?

JArguello-WMF closed subtask T324576: Flink on Kubernetes Helm charts as Resolved.Jan 27 2023, 8:13 PM
dcausse moved this task from Incoming to In Progress on the Discovery-Search (Current work) board.Jan 30 2023, 9:08 AM
dcausse added a subtask: T328675: Create a dse-k8s service demonstrating how to run the rdf-streaming-updater using the flink-app chart.Feb 2 2023, 3:04 PM
dcausse updated the task description. (Show Details)
dcausse updated the task description. (Show Details)Feb 2 2023, 3:07 PM
Ottomata updated the task description. (Show Details)Feb 2 2023, 3:19 PM
JMeybohm updated the task description. (Show Details)Feb 2 2023, 3:41 PM
JMeybohm added a project: serviceops-radar.
Restricted Application added a project: [DEPRECATED] wdwb-tech. · View Herald TranscriptFeb 2 2023, 3:42 PM
• MPhamWMF assigned this task to dcausse.Feb 6 2023, 4:34 PM
gerritbot added a comment.Feb 8 2023, 1:27 PM

dcausse merged https://gitlab.wikimedia.org/repos/search-platform/flink-rdf-streaming-updater/-/merge_requests/1

Initial commit

Maintenance_bot removed a project: Patch-For-Review.Feb 8 2023, 1:30 PM
gerritbot added a comment.Feb 8 2023, 2:43 PM

dcausse opened https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/15

Add repos/search-platform/flink-rdf-streaming-updater

gerritbot added a project: Patch-For-Review.Feb 8 2023, 2:43 PM
gerritbot added a comment.Feb 8 2023, 3:04 PM

jelto merged https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/15

Add repos/search-platform/flink-rdf-streaming-updater

Maintenance_bot removed a project: Patch-For-Review.Feb 8 2023, 3:31 PM
JMeybohm mentioned this in T326729: Remove the .Values.kubernetesApi hack.Mar 8 2023, 1:52 PM
Gehel closed subtask T328675: Create a dse-k8s service demonstrating how to run the rdf-streaming-updater using the flink-app chart as Resolved.Apr 7 2023, 9:40 AM
Gehel set the point value for this task to 0.May 1 2023, 3:28 PM
dcausse moved this task from In Progress to Blocked/Waiting on the Discovery-Search (Current work) board.May 22 2023, 3:23 PM
Gehel closed subtask T289836: Upgrade the WDQS streaming updater to latest flink (1.16) as Resolved.Jun 2 2023, 9:52 AM
dcausse removed dcausse as the assignee of this task.Jun 26 2023, 3:16 PM
dcausse updated the task description. (Show Details)
dcausse updated the task description. (Show Details)Jun 26 2023, 3:18 PM
Gehel added a project: Data-Platform-SRE.Jun 27 2023, 12:55 PM
Gehel moved this task from Incoming to Blocked / Waiting on the Data-Platform-SRE board.
Gehel added a parent task: T340548: [EPIC] Deployment of the Search Update Pipeline on Flink / k8s.Jun 27 2023, 1:25 PM
dcausse added a parent task: T341792: Provision Zookeeper Cluster for storing Flink HA data.Jul 17 2023, 3:20 PM
Gehel subscribed.Jul 17 2023, 3:21 PM

This is blocked by T341792 at the moment.

dcausse removed a parent task: T341792: Provision Zookeeper Cluster for storing Flink HA data.Jul 17 2023, 6:11 PM
dcausse added a subtask: T341792: Provision Zookeeper Cluster for storing Flink HA data.
bking added a subtask: T344614: Add Zookeeper config to 'rdf-streaming-updater' test service on DSE cluster.Aug 21 2023, 2:33 PM
bking claimed this task.Aug 30 2023, 2:22 PM
bking updated the task description. (Show Details)Sep 5 2023, 1:51 PM
Gehel added a subtask: T345668: Find/fix logstash logging for rdf-streaming-updater.Sep 15 2023, 12:40 PM
bking closed subtask T341792: Provision Zookeeper Cluster for storing Flink HA data as Resolved.Sep 19 2023, 4:40 PM
bking added a subtask: T342149: Test common operations in the flink operator/k8s/Flink ZK environment.Sep 20 2023, 1:40 PM
Gehel closed subtask T344614: Add Zookeeper config to 'rdf-streaming-updater' test service on DSE cluster as Resolved.Sep 20 2023, 1:43 PM
Gehel removed a subtask: T345668: Find/fix logstash logging for rdf-streaming-updater.Sep 20 2023, 1:48 PM
bking updated the task description. (Show Details)Sep 21 2023, 6:18 PM
bking updated the task description. (Show Details)Sep 21 2023, 8:04 PM
bking updated the task description. (Show Details)Sep 22 2023, 3:55 PM
bking changed the status of subtask T342149: Test common operations in the flink operator/k8s/Flink ZK environment from Open to In Progress.Oct 5 2023, 2:59 PM
bking closed subtask T342149: Test common operations in the flink operator/k8s/Flink ZK environment as Resolved.Oct 5 2023, 3:12 PM
bking reopened subtask T342149: Test common operations in the flink operator/k8s/Flink ZK environment as In Progress.Oct 12 2023, 10:35 PM
Gehel removed a parent task: T340548: [EPIC] Deployment of the Search Update Pipeline on Flink / k8s.Oct 16 2023, 3:35 PM

Removing this as a dependency of deploying the Search Update Pipeline. We have proven that things work enough. This migration will still need to happen, but can be done independently from the Search Update Pipeline.

bking updated the task description. (Show Details)Oct 17 2023, 1:12 PM
bking added a subtask: T349095: Migrate staging rdf-streaming-updater to flink operator.Oct 17 2023, 1:34 PM
Gehel moved this task from Blocked / Waiting to In Progress on the Data-Platform-SRE board.Oct 17 2023, 3:34 PM
JMeybohm mentioned this in T300033: Use cert-manager for service-proxy certificate creation.Oct 20 2023, 10:06 AM
bking added a subtask: T346456: Improve concurrency limits configuration of the wdqs updater.Oct 27 2023, 3:46 PM
bking removed a subtask: T346456: Improve concurrency limits configuration of the wdqs updater.Oct 27 2023, 4:01 PM
Gehel triaged this task as Medium priority.Nov 3 2023, 10:28 AM
Gehel added a subtask: T350784: Identify/complete post-migration tasks after rdf-streaming-updater migrates to flink operator.Nov 13 2023, 4:22 PM
bking updated the task description. (Show Details)Nov 20 2023, 5:40 PM
bking updated the task description. (Show Details)Nov 29 2023, 4:21 PM
bking updated the task description. (Show Details)Nov 29 2023, 9:42 PM
bking updated the task description. (Show Details)Nov 29 2023, 9:59 PM
bking added a comment.Nov 29 2023, 10:03 PM

I'm happy to say the flink operator migration is complete. Commons and wikidata are stable in both CODFW and EQIAD. As such, I'm resolving this ticket. Post-migration cleanup work continues in T350784 .

bking closed this task as Resolved.Nov 29 2023, 10:03 PM
bking moved this task from In Progress to Done on the Data-Platform-SRE board.
bking closed subtask T349095: Migrate staging rdf-streaming-updater to flink operator as Resolved.Nov 30 2023, 11:24 PM
Gehel closed subtask T342149: Test common operations in the flink operator/k8s/Flink ZK environment as Resolved.Dec 8 2023, 9:06 AM
gerritbot added a comment.Mar 28 2024, 2:56 PM

Change #1015343 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/deployment-charts@master] Remove flink RBAC snowflakes

https://gerrit.wikimedia.org/r/1015343

gerritbot added a project: Patch-For-Review.Mar 28 2024, 2:56 PM
gerritbot added a comment.Apr 8 2024, 11:59 AM

Change #1015343 merged by jenkins-bot:

[operations/deployment-charts@master] Remove flink RBAC snowflakes

https://gerrit.wikimedia.org/r/1015343

gerritbot added a comment.Apr 8 2024, 12:33 PM

Change #1017789 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/deployment-charts@master] Revert "Remove flink RBAC snowflakes"

https://gerrit.wikimedia.org/r/1017789

gerritbot added a comment.Apr 8 2024, 12:33 PM

Change #1017789 merged by JMeybohm:

[operations/deployment-charts@master] Revert "Remove flink RBAC snowflakes"

https://gerrit.wikimedia.org/r/1017789

gerritbot added a comment.Apr 9 2024, 9:39 AM

Change #1018214 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/deployment-charts@master] Revert "Revert "Remove flink RBAC snowflakes""

https://gerrit.wikimedia.org/r/1018214

gerritbot added a comment.Apr 9 2024, 9:49 AM

Change #1018214 merged by jenkins-bot:

[operations/deployment-charts@master] Revert "Revert "Remove flink RBAC snowflakes""

https://gerrit.wikimedia.org/r/1018214

bking closed subtask T350784: Identify/complete post-migration tasks after rdf-streaming-updater migrates to flink operator as Resolved.Jul 23 2024, 7:48 PM
Maintenance_bot removed a project: Patch-For-Review.Jul 23 2024, 8:31 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits