Document ideas & investigation results from out spike with "Spark on k8s" [SPIKE - 1.5 Sprints]
Closed, ResolvedPublic5 Estimated Story Points
Actions

Assigned To

Authored By

	JAllemandou
	Sep 26 2022, 11:52 AM

Description

Problem statement

Spark on K8 currently does not match the features we have through Spark through YARN. Specifically we might we run into issues with Resource Management, Network/Storage bottlenecks and security management.

Primary Task

Documenting our technical explanations, or discussions about some of the challenges we will run into when changing how we interface with Spark and clarify the tradeoffs we will have to contend with.

Research Areas:

Network & Storage Constraints - Decoupling
How Scheduling works
Security Model

Related Objects

Mentioned In: T320812: [SPIKE] Deploy event driven stateless Flink service to DSE cluster
T318712: Enable spark jobs on the dse-k8s cluster via the spark-operator
T317517: DSE Kubernetes Application deployment tutorial

Event Timeline

JAllemandou created this task.Sep 26 2022, 11:52 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 26 2022, 11:52 AM

BTullis subscribed.Sep 26 2022, 11:54 AM

BTullis mentioned this in T317517: DSE Kubernetes Application deployment tutorial.Sep 27 2022, 11:41 AM

• EChetty moved this task from Backlog to To be discussed on the Shared-Data-Infrastructure board.Sep 27 2022, 12:51 PM

Should we start with a link dump of sorts, here? Or are you thinking of a more formal document?

It strikes me that there are at least a couple of strands of work, to begin with, such as:

What do we think Spark on Kubernetes might look like at the WMF?
What potential challenges and/or risks might we face if we implement it as currently envisaged?

Once we identify the potential risk areas (the known unknowns), then perhaps we could think about short and long-term approaches to mitigate them.

I can start by sharing some of my current thinking for point 1 above as well, if you like? More than happy to discuss any of your suggestions and concerns areas too.

BTullis mentioned this in T318712: Enable spark jobs on the dse-k8s cluster via the spark-operator.Sep 27 2022, 3:25 PM

• EChetty renamed this task from Document ideas on "Spark on k8s" to Document ideas & investigation results from out spike with "Spark on k8s" [SPIKE - 1 Sprint].Sep 29 2022, 12:13 PM

• EChetty updated the task description. (Show Details)

• EChetty set the point value for this task to 5.

• EChetty moved this task from To be discussed to Sprint 02 on the Shared-Data-Infrastructure board.Sep 29 2022, 12:17 PM

• EChetty edited projects, added Shared-Data-Infrastructure (Sprint 02); removed Shared-Data-Infrastructure.

• EChetty moved this task from Next Up to In Progress on the Shared-Data-Infrastructure (Sprint 02) board.

• EChetty updated the task description. (Show Details)Oct 10 2022, 12:59 PM

• EChetty renamed this task from Document ideas & investigation results from out spike with "Spark on k8s" [SPIKE - 1 Sprint] to Document ideas & investigation results from out spike with "Spark on k8s" [SPIKE - 1.5 Sprints].Oct 12 2022, 12:46 PM

• EChetty updated the task description. (Show Details)Oct 17 2022, 12:40 PM

• EChetty moved this task from Sprint 02 to Sprint 03 on the Shared-Data-Infrastructure board.Oct 18 2022, 1:25 PM

• EChetty edited projects, added Shared-Data-Infrastructure (Sprint 03); removed Shared-Data-Infrastructure (Sprint 02).

• EChetty moved this task from Next Up to In Progress on the Shared-Data-Infrastructure (Sprint 03) board.

• EChetty updated the task description. (Show Details)Oct 19 2022, 12:41 PM

Document for review here: https://docs.google.com/document/d/1xl1Q1XXcEiHrOVjjdCdauGOLwR7q-uPaD5kWx3YI5D4

JAllemandou moved this task from In Progress to In Review on the Shared-Data-Infrastructure (Sprint 03) board.Oct 21 2022, 1:57 PM

• EChetty updated the task description. (Show Details)Oct 24 2022, 12:30 PM

gmodena mentioned this in T320812: [SPIKE] Deploy event driven stateless Flink service to DSE cluster.Oct 24 2022, 12:48 PM

• EChetty moved this task from Sprint 03 to EQ2 Kanban (Sprints 04-07) on the Shared-Data-Infrastructure board.Nov 8 2022, 10:40 AM

• EChetty edited projects, added Shared-Data-Infrastructure (EQ2 Kanban (Sprints 04-07)); removed Shared-Data-Infrastructure (Sprint 03).

• EChetty moved this task from Next Up to Done on the Shared-Data-Infrastructure (EQ2 Kanban (Sprints 04-07)) board.