Page MenuHomePhabricator
Create Task
Maniphest T351722

Create a helm chart for the spark-history service
Closed, ResolvedPublic
Actions

Description

We need a new chart in operations/deployment-charts for deploying the Spark History Service to Kubernetes.

There is some background information on our use of helm charts here: https://wikitech.wikimedia.org/wiki/Kubernetes/Deployment_Charts

I believe that the README file here is up-to-date in terms of adding a new service:
https://gerrit.wikimedia.org/r/plugins/gitiles/operations/deployment-charts/%2B/refs/heads/master/README.md

We will need to supply a few configuration values and, crucially, a kerberos keytab, so that this service can read from and write to HDFS.

We will be using the spark images here: https://gerrit.wikimedia.org/r/c/operations/docker-images/production-images/+/896363 and running it as a daemon using the history option to the entrypoint.sh script.

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
spark-history: Add missing service template includeoperations/deployment-chartsmaster+2 -1
Define the spark-history chartoperations/deployment-chartsmaster+2 K -0
dse-k8s limitrange: ensure pod max memory is higher than container max memoryoperations/deployment-chartsmaster+3 -0
Mention the fact that 'history' is a valid argument in the error messageoperations/docker-images/production-imagesmaster+3 -3
Customize query in gerrit

Related Objects
Search...

Event Timeline

BTullis created this task.Nov 21 2023, 2:35 PM
BTullis triaged this task as High priority.
BTullis mentioned this in T330176: [Data Platform] Deploy Spark History Service.Nov 21 2023, 2:42 PM
Gehel moved this task from Incoming to Quarterly Goals on the Data-Platform-SRE board.Nov 22 2023, 9:24 AM
BTullis removed BTullis as the assignee of this task.Nov 22 2023, 10:25 AM
brouberol subscribed.Nov 24 2023, 1:28 PM

We can probably take inspiration from https://artifacthub.io/packages/helm/cloudnativeapp/spark-history-server as well

brouberol claimed this task.Nov 27 2023, 9:31 AM
gerritbot added a comment.Nov 28 2023, 8:44 AM

Change 977994 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/docker-images/production-images@master] Mention the fact that 'history' is a valid argument in the error message

https://gerrit.wikimedia.org/r/977994

gerritbot added a project: Patch-For-Review.Nov 28 2023, 8:44 AM
gerritbot added a comment.Nov 29 2023, 4:14 PM

Change 978629 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/deployment-charts@master] Define the spark-history chart

https://gerrit.wikimedia.org/r/978629

brouberol moved this task from Quarterly Goals to In Progress on the Data-Platform-SRE board.Nov 30 2023, 1:49 PM
gerritbot added a comment.Nov 30 2023, 2:17 PM

Change 977994 merged by Brouberol:

[operations/docker-images/production-images@master] Mention the fact that 'history' is a valid argument in the error message

https://gerrit.wikimedia.org/r/977994

brouberol added a comment.Dec 1 2023, 9:55 AM

The helm chart is _mostly_ done. @BTullis @Antoine_Quhen whenever possible and convenient, could I get your opinion on the general design and the default values being used? A lot of hadoop/hdfs config (ultimately rendered into core-site.xml and hdfs-site.xml) were cargo-culted and many might just be irrelevant. Thanks!

Gehel edited projects, added Data-Platform-SRE (2023.12.01 - 2023.12.31); removed Patch-For-Review, Data-Platform-SRE.Dec 6 2023, 9:27 AM
Gehel moved this task from Backlog to In Progress on the Data-Platform-SRE (2023.12.01 - 2023.12.31) board.Dec 6 2023, 9:28 AM
brouberol moved this task from In Progress to Needs Review on the Data-Platform-SRE (2023.12.01 - 2023.12.31) board.Dec 7 2023, 4:17 PM
Stashbot added a comment.Dec 12 2023, 1:45 PM

Mentioned in SAL (#wikimedia-operations) [2023-12-12T13:45:33Z] <brouberol> increasing max container memory requests in dse-k8s from 3GB to 8GB - T351722

gerritbot added a comment.Dec 13 2023, 8:49 AM

Change 982762 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/deployment-charts@master] dse-k8s limitrange: ensure pod max memory is higher than container max memory

https://gerrit.wikimedia.org/r/982762

gerritbot added a project: Patch-For-Review.Dec 13 2023, 8:49 AM
gerritbot added a comment.Dec 13 2023, 9:22 AM

Change 982762 merged by Brouberol:

[operations/deployment-charts@master] dse-k8s limitrange: ensure pod max memory is higher than container max memory

https://gerrit.wikimedia.org/r/982762

Stashbot added a comment.Dec 13 2023, 9:24 AM

Mentioned in SAL (#wikimedia-operations) [2023-12-13T09:24:59Z] <brouberol> increasing pod max requested memory to a higher value than the container max requested memory for dse-k8s-eqiad - T351722

gerritbot added a comment.Dec 15 2023, 8:24 AM

Change 978629 merged by Brouberol:

[operations/deployment-charts@master] Define the spark-history chart

https://gerrit.wikimedia.org/r/978629

brouberol closed this task as Resolved.Dec 15 2023, 8:26 AM
Gehel moved this task from Needs Review to Done on the Data-Platform-SRE (2023.12.01 - 2023.12.31) board.Dec 15 2023, 9:06 AM
gerritbot added a comment.Dec 15 2023, 2:35 PM

Change 983422 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/deployment-charts@master] spark-history: Add missing service template include

https://gerrit.wikimedia.org/r/983422

gerritbot added a comment.Dec 15 2023, 2:42 PM

Change 983422 merged by Brouberol:

[operations/deployment-charts@master] spark-history: Add missing service template include

https://gerrit.wikimedia.org/r/983422

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits