Maniphest T432322

Move production spark jobs from YARN to k8s
Open, Needs TriagePublic
Actions

Assigned To

Authored By

	JAllemandou
	Jul 16 2026, 9:15 AM

Project Tags

Referenced Files

None

Subscribers

Description

This ticket is an epic covering the work that needs to happen to move data-engineering spark jobs runtime from YARN to k8s.
We'll use it as a parent task to organize the work.

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Open		BTullis	T435461 [Epic] Migrate the Data Lake from Hadoop to Kubernetes and Ceph/S3
		Open		JAllemandou	T432322 Move production spark jobs from YARN to k8s
		Open		JAllemandou	T412925 [Infra] Carry out end-user testing of spark on kubernetes
		Open		JAllemandou	T297231 [Infra] Mechanism to get Apache Spark operational metrics
		Resolved		ayounsi	T304001 Allow access to prometheus-pushgateway.discovery.wmnet port 80 from within Analytics VLAN
		Open		None	T413977 Deploy Kyuubi to enable dynamic spark-sql clusters in dse-k8s

Event Timeline

JAllemandou created this task.Jul 16 2026, 9:15 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 16 2026, 9:15 AM

JAllemandou added a project: Epic.Jul 16 2026, 9:17 AM

JAllemandou added a subtask: T412925: [Infra] Carry out end-user testing of spark on kubernetes.

JAllemandou added a subtask: T297231: [Infra] Mechanism to get Apache Spark operational metrics.

BTullis added a parent task: T435461: [Epic] Migrate the Data Lake from Hadoop to Kubernetes and Ceph/S3.Thu, Aug 20, 4:12 PM

BTullis added a subtask: T413977: Deploy Kyuubi to enable dynamic spark-sql clusters in dse-k8s.Fri, Aug 21, 12:12 PM