Maniphest T324578

[EPIC] Flink Applications on Kubernetes
Closed, ResolvedPublic
Actions

Description

This is a parent task for being able to deploy and manage Flink applications in Kubernetes at WMF.

We are targeting application deployment mode only, meaning each Flink deployment runs a single application, as opposed to 'session' mode, where a Flink deployment cluster can run multiple applications.

Flink and Flink Operator images - T316519: Flink application and flink-kubernetes-operator production docker images
Flink Operator and Flink Application helm charts - T324576: Flink on Kubernetes Helm charts
Deploy and test stateless application in DSE cluster - T320812: [SPIKE] Deploy event driven stateless Flink service to DSE cluster (mediawiki enrichment pipelines)
Deploy and test stateful application in DSE cluster - T321491: Evaluate Flink Operator on DSE Kubernetes Cluster for deployment and management of stateful search applications (rdf-streaming-updater)

...

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Resolved		• lbowmaker	T324578 [EPIC] Flink Applications on Kubernetes
		Resolved		Ottomata	T316519 Flink application and flink-kubernetes-operator production docker images
		Resolved		gmodena	T320812 [SPIKE] Deploy event driven stateless Flink service to DSE cluster
		Resolved		BTullis	T321682 Create kubernetes namespace and user for the stream_enrichment PoC project
		Resolved		Ottomata	T324576 Flink on Kubernetes Helm charts
		Resolved		Ottomata	T326252 k8s deployment-charts mesh module should allow use of mesh without public_port Service
		Duplicate		bking	T321491 Evaluate Flink Operator on DSE Kubernetes Cluster for deployment and management of stateful search applications
		Resolved		bking	T321587 Build Flink minikube playground on WMCS Search project

Event Timeline

Ottomata created this task.Dec 6 2022, 3:33 PM

Restricted Application added a project: Data-Engineering. · View Herald TranscriptDec 6 2022, 3:33 PM

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

Ottomata added subtasks: T316519: Flink application and flink-kubernetes-operator production docker images, T320812: [SPIKE] Deploy event driven stateless Flink service to DSE cluster, T324576: Flink on Kubernetes Helm charts, T321491: Evaluate Flink Operator on DSE Kubernetes Cluster for deployment and management of stateful search applications.Dec 6 2022, 3:34 PM

Ottomata updated the task description. (Show Details)Dec 6 2022, 3:36 PM

JMeybohm subscribed.Dec 7 2022, 8:49 AM

xcollazo subscribed.Dec 7 2022, 3:36 PM

Ottomata updated the task description. (Show Details)Dec 9 2022, 10:50 PM

Ottomata moved this task from Backlog to Parent Tasks/Epics on the Event-Platform board.Dec 13 2022, 8:16 PM

Ottomata updated the task description. (Show Details)Dec 14 2022, 5:10 PM

• EChetty edited projects, added Data-Engineering-Planning; removed Data-Engineering.Jan 6 2023, 11:01 AM

• EChetty moved this task from Backlog to Event Platform on the Data-Engineering-Planning board.Jan 6 2023, 12:45 PM

Ottomata updated the task description. (Show Details)Jan 11 2023, 6:27 PM

JArguello-WMF closed subtask T324576: Flink on Kubernetes Helm charts as Resolved.Jan 27 2023, 8:13 PM

JArguello-WMF closed subtask T316519: Flink application and flink-kubernetes-operator production docker images as Resolved.

Ottomata updated the task description. (Show Details)Mar 29 2023, 4:00 PM

Ottomata updated the task description. (Show Details)

Ottomata closed subtask T320812: [SPIKE] Deploy event driven stateless Flink service to DSE cluster as Resolved.

JArguello-WMF removed a project: Data-Engineering-Planning.Jun 29 2023, 9:48 PM

Restricted Application added a project: Data-Engineering. · View Herald TranscriptJun 29 2023, 9:48 PM

JArguello-WMF moved this task from Incoming (new tickets) to Event Platform Backlog on the Data-Engineering board.Jun 29 2023, 10:31 PM

JArguello-WMF edited projects, added Data Engineering and Event Platform Team; removed Data-Engineering.Jun 30 2023, 4:15 PM

Restricted Application added a project: Data-Engineering. · View Herald TranscriptJun 30 2023, 4:15 PM

JArguello-WMF moved this task from Data Eng Backlog to Parent Tasks/Epics on the Data Engineering and Event Platform Team board.Jun 30 2023, 4:17 PM

JArguello-WMF removed a project: Data-Engineering.Jun 30 2023, 4:18 PM

Restricted Application added a project: Data-Engineering. · View Herald TranscriptJun 30 2023, 4:18 PM

• lbowmaker closed this task as Resolved.Oct 20 2023, 5:00 PM

• lbowmaker claimed this task.