We now have the ability to create //spark-enabled// namespaces in Kubernetes.
This is achieved in two ways:
# Add this namespace to the list of [[https://github.com/wikimedia/operations-deployment-charts/blob/master/helmfile.d/admin_ng/spark-operator/values-production.yaml|jobNamespaces]] that are monitored by the [[https://github.com/wikimedia/operations-deployment-charts/tree/master/charts/spark-operator|spark-operator]].
# Deploy an instance of the [[https://github.com/wikimedia/operations-deployment-charts/tree/master/charts/spark-support|spark-support]] chart to this namespace.
What this means in practice is:
* The spark-operator is watching this namespace and will handle any `SparkApplication` or `SheduledSparkApplication` objects that are created.
* A `spark` serviceaccount exists, which has [[https://github.com/wikimedia/operations-deployment-charts/blob/master/charts/spark-support/templates/rbac.yaml|elevated privileges]] within the namespace.
* Full access to manage pods, services, configmaps, secrets, and persistent volume claims
* We can permit serviceaccounts in other namespaces to create pods etc.
* Typically, this will be an `airflow` serviceaccount in an airflow instance namespace.
* Firewall rules are in place, permitting spark pods access to Kerberos, Hive, HDFS, and other services, as required.
* A kerberos keytab has been deployed as a secret, which can be used to authenticate to Hive and HDFS.
We currently have the `analytics-test` namespace fully enabled in this way and it would be helpful for end users to start using spark and providing feedback to help guide the next steps on configuration.
In order to gain access to the namespace, users will want to run the following from a deployment server.
```
kube-env analytics-test-deploy dse-k8s-eqiad
```
These credentials are only accessible to members of the [[https://github.com/wikimedia/operations-puppet/blob/production/modules/admin/data/data.yaml#L872-L875|analytics-deployers]] group, which is effectively an alias for the `analytics-admins` group. See [[https://wikitech.wikimedia.org/wiki/Data_Platform/Data_access#Analytics_shell_groups_explained|this page]] for more information on the shell groups.
The simplest way to get started testing is to make use of the `spark-toolbox` [[https://github.com/wikimedia/operations-deployment-charts/blob/master/charts/spark-support/templates/_deplopyment.yaml.tpl#L1|deployment]] that is defined within the `spark-support chart.
```
btullis@deploy2002:~$ kubectl get deployments spark-toolbox
NAME READY UP-TO-DATE AVAILABLE AGE
spark-toolbox 3/3 3 3 42h
```
This is a container that uses the [[https://gitlab.wikimedia.org/repos/data-engineering/spark|spark image]] and has all of the correct config files, secrets, environment variables etc. but is just running a `sleep infinity` command, ready for a user to start a shell and run interactive commands.
We can launch a shell in this toolbox like this:
```
kubectl exec -it spark-toolbox-6fcf874549-n4dff -- /entrypoint.sh bash
```
The first suffix for the pod will change on each deployment, and the second will be unique to each of the toolbox replicas.
Once inside the toolbox pod, we can start to launch spark sessions in both `client` and `cluster` mode.
Client mode sessions include the use of: `spark-sql`, `spark-shell`, and `pyspark`, as well as `spark-submit`.
Cluster mode sessions just use `spark-submit`.
When running a client mode setting, there is only one required command-line option, which is `spark.driver.host`.
In our case, we want to set this value to either the IPv4 or the IPv6 address of the pod.
So the following should all work for starting a spark cluster in client mode.
```
spark-sql --conf spark.driver.host=[$(hostname -i|awk '{print $1}')]
spark-shell --conf spark.driver.host=[$(hostname -i|awk '{print $1}')]
pyspark --conf spark.driver.host=[$(hostname -i|awk '{print $1}')]
```
In cluster mode, we do not have to override this value.
We have seen the following sparkPi job succeed from a toolbox pod.
```
spark-submit --verbose --deploy-mode cluster --class org.apache.spark.examples.SparkPi --name "Spark Pi" local:///opt/spark/examples/jars/spark-examples_2.12-3.5.7.jar
```
The executor pods are created and deleted dynamically, but the driver pod remains in a `completed` state, so we can check the logs.
```
btullis@deploy2002:~$ kubectl logs spark-pi-a6237b9b2bcbf64a-driver | tail -n 1
Pi is roughly 3.1441357206786034
```
There are certain things that we will want to be aware of and to carry on configuring, including:
* [[https://spark.apache.org/docs/latest/running-on-kubernetes.html#using-kubernetes-volumes|Use of kubernetes volumes]], including:
* Use of [[https://spark.apache.org/docs/latest/running-on-kubernetes.html#local-storage|local storage]] on the dse-k8s-worker nodes.
* [[https://spark.apache.org/docs/latest/running-on-kubernetes.html#dependency-management|Dependency management]] using HDFS and/or S3 sources.
* Accessing the [[https://spark.apache.org/docs/latest/running-on-kubernetes.html#accessing-driver-ui|driver UI]].
* [[https://spark.apache.org/docs/latest/running-on-kubernetes.html#ipv4-and-ipv6|Dual-stack]] networking options.
If we find that there are additional options that we wish to add to the spark or hadoop config files, these will be made to the `spark-support` chart and the [[https://github.com/wikimedia/operations-deployment-charts/blob/master/helmfile.d/dse-k8s-services/analytics-test/values.yaml|values.yaml]] file specific to each namespace.
Any feedback on the functionality and testing is welcome.