Add means to upgrade the flink code even when incompatible serialization changes are involved
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	dcausse
	Mar 8 2021, 8:58 AM

Description

As wdqs maintainer I want a way to tranform a savepoint (drained) to a CSV file so that I can reuse the existing bootstrap job to resume a pipeline even when incompatible serialization changes are made.

add a job that
- dumps a CSV file similar to the one created by org.wikidata.query.rdf.spark.EntityRevisionMapGenerator
- dumps another CSV file kafka consumer offsets
adapt the UpdaterBootstrapJob to support setting consumer offsets

AC:

the pipeline can always be upgraded using this procedure:
1. [old code]: stop&drain the pipeline storing a savepoint
2. [old code]: tranform the savepoint to a set of CSV files
3. [new code]: run the bootstrap job with the CSV files
4. [new code]: resume the pipeline

Details

	Subject	Repo	Branch	Lines +/-
	Add a state extraction job	wikidata/query/rdf	master	+535 -62

Customize query in gerrit

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Resolved		Gehel	T244590 [Epic] Rework the WDQS updater as an event driven application
		Resolved		dcausse	T276750 Add means to upgrade the flink code even when incompatible serialization changes are involved

Event Timeline

dcausse created this task.Mar 8 2021, 8:58 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 8 2021, 8:58 AM

dcausse claimed this task.Mar 8 2021, 9:09 AM

dcausse added projects: Wikidata-Query-Service, Discovery-Search (Current work).

dcausse updated the task description. (Show Details)

Restricted Application added a project: Wikidata. · View Herald TranscriptMar 8 2021, 9:09 AM

dcausse added a parent task: T244590: [Epic] Rework the WDQS updater as an event driven application.Mar 8 2021, 9:09 AM

dcausse moved this task from Incoming to In Progress on the Discovery-Search (Current work) board.Mar 8 2021, 9:13 AM

TJones renamed this task from Add a mean to upgrade the flink code even when incompatible serialization changes are involved to Add means to upgrade the flink code even when incompatible serialization changes are involved.Mar 8 2021, 4:45 PM

MPhamWMF moved this task from Incoming to Current work on the Wikidata-Query-Service board.Mar 8 2021, 4:45 PM

Change 665082 had a related patch set uploaded (by DCausse; owner: DCausse):
[wikidata/query/rdf@master] Add state extraction job

https://gerrit.wikimedia.org/r/665082

gerritbot added a project: Patch-For-Review.Mar 22 2021, 5:55 PM

dcausse moved this task from In Progress to Blocked/Waiting on the Discovery-Search (Current work) board.Mar 23 2021, 7:52 AM

dcausse moved this task from Blocked/Waiting to Needs review on the Discovery-Search (Current work) board.Mar 29 2021, 3:26 PM

Change 665082 merged by jenkins-bot:

[wikidata/query/rdf@master] Add a state extraction job

https://gerrit.wikimedia.org/r/665082

Maintenance_bot removed a project: Patch-For-Review.Apr 13 2021, 1:11 PM

dcausse moved this task from Needs review to To Be Deployed on the Discovery-Search (Current work) board.May 10 2021, 3:35 PM

dcausse moved this task from To Be Deployed to Needs Reporting on the Discovery-Search (Current work) board.May 17 2021, 3:28 PM