Page MenuHomePhabricator

Modify flink-job.py script to handle WCQS streaming updater deployment
Open, Needs TriagePublic3 Estimated Story Points

Description

As a WCQS maintainter I want to be able to deploy WCQS streaming updater the same way as it is being done for WDQS.

Flink job deployment script (flink-job.py) has a definition of envs, but they are defined only with WDQS in mind. We should have environments defined for WCQS as well

Acceptance Criteria:

  • flink-job.py can be used to deploy WCQS streaming updater for any enviroment

Event Timeline

I started looking into this, changing the script is reasonably easy but the correct values to use are a bit more of a mystery. A few questions:

  • We currently have staging, eqiad and codfw environments, how do we want to name them going forward to include wcqs, wcqs-eqiad? Is there any pain involved if we change the names of deployed wdqs envs to match?
  • Does the wcqs variant need it's own object storage, or should it use a separate directory in the existing object storage? Currently wdqs-eqiad goes to swift://rdf-streaming-updater-eqiad.thanos-swift/wikidata/checkpoints, Could wcqs go to swift://rdf-streaming-updater-eqiad.thanos-swift/commons/checkpoints?
  • Will wcqs be reusing the helmfile config paths (/etc/helmfile-defaults/general-[staging|eqiad|codfw].yaml), or do we need wcqs specific ones? (i think relatedly, did we agree wcqs updater runs in the same flink cluster, which would suggest these config files will be the same?)
  • More a question, but staging specifys test.wikidata.org as the hostname, but build_http_endpoints() doesn't output an entry mapping it to the proxy, should it?

Change 743042 had a related patch set uploaded (by Ebernhardson; author: Ebernhardson):

[wikidata/query/deploy@master] flink-job.py: Make space for wcqs

https://gerrit.wikimedia.org/r/743042

Change 743043 had a related patch set uploaded (by Ebernhardson; author: Ebernhardson):

[wikidata/query/deploy@master] flink-job.py: Add wcqs deployment

https://gerrit.wikimedia.org/r/743043

Change 743042 merged by DCausse:

[wikidata/query/deploy@master] flink-job.py: Make space for wcqs

https://gerrit.wikimedia.org/r/743042