Maniphest T331971

[dse-k8s] Deploy spark cli to submit jobs on DSE K8S cluster with K8S config
Open, LowPublic
Actions

Assigned To

None

Authored By

	• nfraison
	Mar 14 2023, 10:31 AM

Description

DoD:

cli deployed on all stats/airflow/launcher instance
appropriate K8S Config deployed on stats/airflow/launcher instance based on spark-deploy svc account. Should only be accessible for root user and Data Engineering team as it will only be used for testing spark on K8S

Related Objects
Search...

Status	Assigned	Task
In Progress	None	T327267 Create a DSE Kubernetes cluster with support for persistent storage from Ceph
Duplicate	None	T327258 DSE Experiment - User Story 2 (Make Compute available)
In Progress	None	T318712 Enable spark jobs on the dse-k8s cluster via the spark-operator
Open	None	T331971 [dse-k8s] Deploy spark cli to submit jobs on DSE K8S cluster with K8S config

Event Timeline

• nfraison created this task.Mar 14 2023, 10:31 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 14 2023, 10:31 AM

JArguello-WMF moved this task from Backlog to To be discussed on the Shared-Data-Infrastructure board.Mar 14 2023, 3:53 PM

BTullis added a project: Data-Platform-SRE.Jun 9 2023, 11:56 AM

JArguello-WMF removed a project: Shared-Data-Infrastructure.Jun 29 2023, 1:41 PM

BTullis renamed this task from Deploy spak cli to submit jobs on DSE K8S cluster with K8S config to [dse-k8s] Deploy spark cli to submit jobs on DSE K8S cluster with K8S config.Jul 3 2023, 12:04 PM

BTullis triaged this task as Low priority.

BTullis added a parent task: T318712: Enable spark jobs on the dse-k8s cluster via the spark-operator.Jul 3 2023, 12:08 PM

BTullis merged a task: T318923: Add the sparkctl binary to the stat boxes.

BTullis added a subscriber: • EChetty.

Gehel moved this task from Incoming to Misc on the Data-Platform-SRE board.Dec 7 2023, 2:21 PM

[dse-k8s] Deploy spark cli to submit jobs on DSE K8S cluster with K8S configOpen, LowPublicActions

Description

Related ObjectsSearch...

Event Timeline

[dse-k8s] Deploy spark cli to submit jobs on DSE K8S cluster with K8S config
Open, LowPublic
Actions

Related Objects
Search...