Spark jobs required to have hadoop config (core/hdfs-site.xml files) available to be able to access hdfs
It is possible to push a configmap with this configuration for each job but it will leads to pushing same config multiple times
So we should add common configmap for it (one per hadoop cluster)
| Subject | Repo | Branch | Lines +/- | |
|---|---|---|---|---|
| spark: add hadoop conf configmap | operations/deployment-charts | master | +236 -1 |
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| In Progress | None | T318712 Enable spark jobs on the dse-k8s cluster via the spark-operator | |||
| Resolved | BTullis | T406833 Create a helm chart that manages the resources required for running useful spark jobs on kubernetes | |||
| Resolved | BTullis | T332909 [dse-k8s] Provide common hadooop config for spark jobs |
Change 902402 had a related patch set uploaded (by Nicolas Fraison; author: Nicolas Fraison):
[operations/deployment-charts@master] spark: add hadoop conf configmap
nfraison opened https://gitlab.wikimedia.org/repos/data-engineering/spark8s/-/merge_requests/3
add hadoop configmap volume to spark pods
Resetting task status from "In Progress" to "Open" as this task has been "in progress" for more than two years.
Change #902402 abandoned by Btullis:
[operations/deployment-charts@master] spark: add hadoop conf configmap
Reason:
Now taking a different approach