Maniphest T326328

Create helm charts for the cirrus-streaming-updater flink jobs
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	dcausse
	Jan 5 2023, 2:53 PM

Tags

Referenced Files

None

Subscribers

Description

Following up on the work done in T324576 we should create helm charts in the deployment-charts repo to deploy the cirrus-streaming-updater flink jobs to the wikikube k8s cluster.

The chart should the features provided by the flink operator (i.e. FlinkDeployment CRD).

The service should access:

thanos-swift using the S3 compat layer
kafka-main and/or kafka-test clusters
an elasticsearch cluster for testing
mediawiki API (using api-ro for testing low volume wikis) but an async MW cluster should be used instead (existing jobrunners might be usable, ref: T317283#8220470).

AC:

a chart using the flink-k8s-operator is added to deployment-charts
the flink jobs are deployed to the wikikube staging cluster
updates from testwiki are propagated to a test elasticsearch cluster

Details

	Subject	Repo	Branch	Lines +/-
	cirrus streaming updater service	operations/deployment-charts	master	+436 -0

Customize query in gerrit

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Open		None	T317045 [Epic] Re-architect the Search Update Pipeline
		Resolved		EBernhardson	T326328 Create helm charts for the cirrus-streaming-updater flink jobs

Event Timeline

dcausse created this task.Jan 5 2023, 2:53 PM

Restricted Application added a project: Discovery-Search. · View Herald TranscriptJan 5 2023, 2:53 PM

Gehel moved this task from needs triage to Current work on the Discovery-Search board.Jan 9 2023, 4:44 PM

Gehel edited projects, added Discovery-Search (Current work); removed Discovery-Search.

bking subscribed.Jan 10 2023, 4:06 PM

TJones updated the task description. (Show Details)Feb 6 2023, 4:50 PM

MPhamWMF updated the task description. (Show Details)Feb 6 2023, 4:50 PM

MPhamWMF set the point value for this task to 8.

MPhamWMF moved this task from Incoming to Ready for Dev -- SRE/Ops on the Discovery-Search (Current work) board.

dcausse claimed this task.Feb 14 2023, 9:02 AM

dcausse removed dcausse as the assignee of this task.

dcausse moved this task from Ready for Dev -- SRE/Ops to In Progress on the Discovery-Search (Current work) board.

dcausse moved this task from In Progress to Ready for Dev -- SRE/Ops on the Discovery-Search (Current work) board.

Moving out as it seems unlikely that we will be ready this quarter.

bking edited projects, added Discovery-Search; removed Discovery-Search (Current work).May 15 2023, 7:41 PM

Gehel removed the point value for this task.May 22 2023, 12:56 PM

Gehel moved this task from needs triage to ML & Data Pipeline on the Discovery-Search board.

EBernhardson claimed this task.Oct 5 2023, 5:57 PM

EBernhardson edited projects, added Discovery-Search (Current work); removed Discovery-Search.

Change 951960 had a related patch set uploaded (by Ebernhardson; author: Ebernhardson):

[operations/deployment-charts@master] cirrus streaming updater service

https://gerrit.wikimedia.org/r/951960

gerritbot added a project: Patch-For-Review.Oct 6 2023, 4:40 PM

Change 951960 merged by Bking:

[operations/deployment-charts@master] cirrus streaming updater service

https://gerrit.wikimedia.org/r/951960

Maintenance_bot removed a project: Patch-For-Review.Oct 6 2023, 8:12 PM

EBernhardson moved this task from Ready for Dev -- SRE/Ops to Needs Reporting on the Discovery-Search (Current work) board.Nov 16 2023, 6:14 PM

Gehel closed this task as Resolved.Nov 17 2023, 9:40 AM