Upgrade the WDQS streaming updater to latest flink (1.16)
Closed, ResolvedPublic5 Estimated Story Points
Actions

Assigned To

Authored By

	dcausse
	Aug 27 2021, 6:48 AM

Description

Upstream bug we might have seen while running on our k8s setup:

https://issues.apache.org/jira/browse/FLINK-20417

Upstream bug that prevents us from upgrading to 1.13:

https://issues.apache.org/jira/browse/FLINK-23728

Details

Subject	Repo	Branch	Lines +/-
Upgrade to flink 1.16	wikidata/query/flink-rdf-streaming-updater	master	+30 -54
Upgrade to flink 1.16.0	wikidata/query/rdf	master	+44 -47
Upgrade to flink 1.14.6	wikidata/query/rdf	master	+42 -27
Upgrade to flink 1.16.0	wikidata/query/rdf	flink_1_16	+44 -47
Upgrade to flink 1.14.6	wikidata/query/rdf	flink_1_16	+54 -27

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Resolved	Gehel	T244590 [Epic] Rework the WDQS updater as an event driven application
Resolved	bking	T326409 Migrate the wdqs streaming updater flink jobs to flink-k8s-operator deployment model
Resolved	dcausse	T289836 Upgrade the WDQS streaming updater to latest flink (1.16)

Event Timeline

dcausse created this task.Aug 27 2021, 6:48 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 27 2021, 6:48 AM

Maintenance_bot added a project: Wikidata.Aug 27 2021, 7:46 AM

Gehel moved this task from Incoming to Current work on the Wikidata-Query-Service board.Aug 30 2021, 3:18 PM

Gehel added a project: Discovery-Search (Current work).

Gehel set the point value for this task to 5.Aug 30 2021, 3:53 PM

Gehel moved this task from Incoming to Ready for Dev -- SWE on the Discovery-Search (Current work) board.

Gehel added a parent task: T244590: [Epic] Rework the WDQS updater as an event driven application.Sep 1 2021, 2:16 PM

dcausse renamed this task from Upgrade to latest flink (1.13.2) to Upgrade to latest flink (1.14).Sep 7 2021, 5:52 PM

dcausse updated the task description. (Show Details)

dcausse moved this task from Ready for Dev -- SWE to Waiting on the Discovery-Search (Current work) board.

MPhamWMF moved this task from Waiting to Ready for Dev -- SWE on the Discovery-Search (Current work) board.Oct 25 2021, 3:58 PM

Change 719456 had a related patch set uploaded (by DCausse; author: DCausse):

[wikidata/query/rdf@master] Upgrade to flink 1.14.0

https://gerrit.wikimedia.org/r/719456

gerritbot added a project: Patch-For-Review.Oct 28 2021, 2:47 PM

dcausse moved this task from Ready for Dev -- SWE to Waiting on the Discovery-Search (Current work) board.Nov 2 2021, 6:01 PM

Gehel triaged this task as Medium priority.Feb 22 2022, 8:25 PM

Gehel removed projects: Patch-For-Review, Discovery-Search (Current work).

Gehel moved this task from Current work to Scaling on the Wikidata-Query-Service board.

dcausse renamed this task from Upgrade to latest flink (1.14) to Upgrade the WDQS streaming updater to latest flink (1.15).Aug 29 2022, 12:26 PM

dcausse added a subscriber: Event-Platform.

Aklapper added a project: Event-Platform.Aug 30 2022, 6:27 PM

Aklapper removed a subscriber: Event-Platform.

Restricted Application added a project: Data-Engineering. · View Herald TranscriptAug 30 2022, 6:27 PM

• EChetty edited projects, added Data-Engineering-Planning; removed Data-Engineering.Sep 6 2022, 10:41 AM

• EChetty moved this task from Backlog to Radar on the Data-Engineering-Planning board.Nov 7 2022, 11:10 AM

dcausse moved this task from Scaling to Incoming on the Wikidata-Query-Service board.Dec 19 2022, 10:49 AM

dcausse moved this task from Incoming to Scaling on the Wikidata-Query-Service board.

dcausse added a project: Discovery-Search (Current work).

dcausse moved this task from Scaling to Current work on the Wikidata-Query-Service board.

dcausse added a parent task: T326409: Migrate the wdqs streaming updater flink jobs to flink-k8s-operator deployment model.Jan 6 2023, 1:47 PM

bking subscribed.Jan 11 2023, 3:14 PM

dcausse claimed this task.Jan 12 2023, 2:15 PM

dcausse moved this task from Waiting to In Progress on the Discovery-Search (Current work) board.

dcausse renamed this task from Upgrade the WDQS streaming updater to latest flink (1.15) to Upgrade the WDQS streaming updater to latest flink (1.16).Jan 13 2023, 3:37 PM

Change 879822 had a related patch set uploaded (by DCausse; author: DCausse):

[wikidata/query/rdf@master] Upgrade to flink 1.16.0

https://gerrit.wikimedia.org/r/879822

gerritbot added a project: Patch-For-Review.Jan 13 2023, 3:38 PM

Change 881893 had a related patch set uploaded (by DCausse; author: DCausse):

[wikidata/query/flink-rdf-streaming-updater@master] Upgrade to flink 1.16

https://gerrit.wikimedia.org/r/881893

Change 882748 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] dse-k8s: add rdf-streaming-update-ng namespace

https://gerrit.wikimedia.org/r/882748

dcausse mentioned this in T326409: Migrate the wdqs streaming updater flink jobs to flink-k8s-operator deployment model.Jan 24 2023, 2:04 PM

Change 882748 abandoned by Bking:

[operations/puppet@production] dse-k8s: add rdf-streaming-update-ng namespace

Reason:

Not needed, we will use the current 'rdf-streaming-updater' namespace

https://gerrit.wikimedia.org/r/882748

Change 882748 restored by Bking:

[operations/puppet@production] dse-k8s: add rdf-streaming-update-ng namespace

https://gerrit.wikimedia.org/r/882748

The updater with flink 1.16 did run for about 3 days in yarn, notable change is that it required 5g of mem to run without failure during backfills. Current task_manager limits on k8s is at 2500M so this means doubling this value for both data centers and both jobs (~15G in each DC). There might be lower values that work but it might require tuning every components individually so if the mem is available I'd be for just increasing the mem limits.

Change 886116 had a related patch set uploaded (by DCausse; author: DCausse):

[wikidata/query/rdf@flink_1_16] Upgrade to flink 1.14.6

https://gerrit.wikimedia.org/r/886116

Change 886117 had a related patch set uploaded (by DCausse; author: DCausse):

[wikidata/query/rdf@flink_1_16] Upgrade to flink 1.16.0

https://gerrit.wikimedia.org/r/886117

Change 719456 abandoned by DCausse:

[wikidata/query/rdf@master] Upgrade to flink 1.14.6

Reason:

moved to the flink_1_16 branch

https://gerrit.wikimedia.org/r/719456

Change 879822 abandoned by DCausse: