[Search Update Pipeline] Set max parallelism explicitly on operators with a state
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	dcausse
	Sep 19 2023, 7:21 AM

Description

Max parallelism should be hardcoded in the application and set explicitly on the flink job

Reference: https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/ops/production_ready/

AC:

max parallelism is hard-coded

Details

	Title	Reference	Author	Source Branch	Dest Branch
	Set max parallelism explicitly	repos/search-platform/cirrus-streaming-updater!33	ebernhardson	work/ebernhardson/max-parallelism	main

Customize query in GitLab

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Open		None	T317045 [Epic] Re-architect the Search Update Pipeline
		Resolved		EBernhardson	T346718 [Search Update Pipeline] Set max parallelism explicitly on operators with a state

Event Timeline

dcausse created this task.Sep 19 2023, 7:21 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 19 2023, 7:21 AM

dcausse added a project: Discovery-Search.Sep 19 2023, 7:22 AM

dcausse added a parent task: T317045: [Epic] Re-architect the Search Update Pipeline.

bking renamed this task from [Search Update Pipeline] Set max pallelism explicitely on operators with a state to [Search Update Pipeline] Set max parallelism explicitely on operators with a state.Sep 19 2023, 7:20 PM

Gehel moved this task from needs triage to Current work on the Discovery-Search board.Oct 2 2023, 3:22 PM

Gehel edited projects, added Discovery-Search (Current work); removed Discovery-Search.

TJones renamed this task from [Search Update Pipeline] Set max parallelism explicitely on operators with a state to [Search Update Pipeline] Set max parallelism explicitly on operators with a state.Oct 9 2023, 3:39 PM

TJones updated the task description. (Show Details)

Gehel moved this task from Incoming to Ready for Dev -- SWE on the Discovery-Search (Current work) board.Oct 9 2023, 3:42 PM

Gehel set the point value for this task to 1.

dcausse updated the task description. (Show Details)Oct 9 2023, 3:42 PM

dcausse removed the point value for this task.

EBernhardson claimed this task.Oct 10 2023, 5:41 PM

EBernhardson moved this task from Ready for Dev -- SWE to Needs review on the Discovery-Search (Current work) board.

ebernhardson opened https://gitlab.wikimedia.org/repos/search-platform/cirrus-streaming-updater/-/merge_requests/33

Set max parallelism explicitly

dcausse merged https://gitlab.wikimedia.org/repos/search-platform/cirrus-streaming-updater/-/merge_requests/33

Set max parallelism explicitly

Maintenance_bot removed a project: Patch-For-Review.Oct 11 2023, 9:30 AM