Modify flink-job.py script to handle WCQS streaming updater deployment
Closed, ResolvedPublic3 Estimated Story Points
Actions

Assigned To

Authored By

	• Zbyszko
	Nov 25 2021, 11:04 AM

Description

As a WCQS maintainter I want to be able to deploy WCQS streaming updater the same way as it is being done for WDQS.

Flink job deployment script (flink-job.py) has a definition of envs, but they are defined only with WDQS in mind. We should have environments defined for WCQS as well

Acceptance Criteria:

flink-job.py can be used to deploy WCQS streaming updater for any enviroment

Details

Subject	Repo	Branch	Lines +/-
Add per job config for namespaces	wikidata/query/deploy	master	+12 -2
flink-job.py: Add wcqs deployment	wikidata/query/deploy	master	+22 -0
flink-job.py: Make space for wcqs	wikidata/query/deploy	master	+30 -17

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Resolved	Ladsgroup	T271851 Clean up gui from the wdqs deploy repo and puppet
Resolved	None	T260568 [EPIC] Productionize WCQS
Resolved	EBernhardson	T296468 Modify flink-job.py script to handle WCQS streaming updater deployment

Event Timeline

• Zbyszko created this task.Nov 25 2021, 11:04 AM

MPhamWMF moved this task from Incoming to Current work on the Wikidata-Query-Service board.Nov 29 2021, 4:33 PM

MPhamWMF set the point value for this task to 3.Nov 29 2021, 4:40 PM

MPhamWMF moved this task from Incoming to Ready for Dev -- SWE on the Discovery-Search (Current work) board.

EBernhardson claimed this task.Nov 30 2021, 9:13 PM

EBernhardson moved this task from Ready for Dev -- SWE to In Progress on the Discovery-Search (Current work) board.

I started looking into this, changing the script is reasonably easy but the correct values to use are a bit more of a mystery. A few questions:

We currently have staging, eqiad and codfw environments, how do we want to name them going forward to include wcqs, wcqs-eqiad? Is there any pain involved if we change the names of deployed wdqs envs to match?

Does the wcqs variant need it's own object storage, or should it use a separate directory in the existing object storage? Currently wdqs-eqiad goes to swift://rdf-streaming-updater-eqiad.thanos-swift/wikidata/checkpoints, Could wcqs go to swift://rdf-streaming-updater-eqiad.thanos-swift/commons/checkpoints?

Will wcqs be reusing the helmfile config paths (/etc/helmfile-defaults/general-[staging|eqiad|codfw].yaml), or do we need wcqs specific ones? (i think relatedly, did we agree wcqs updater runs in the same flink cluster, which would suggest these config files will be the same?)

More a question, but staging specifys test.wikidata.org as the hostname, but build_http_endpoints() doesn't output an entry mapping it to the proxy, should it?