We now have the ability to create spark-enabled namespaces in Kubernetes.
This is achieved in two ways:
- Add this namespace to the list of jobNamespaces that are monitored by the spark-operator.
- Deploy an instance of the spark-support chart to this namespace.
What this means in practice is:
- The spark-operator is watching this namespace and will handle any SparkApplication or SheduledSparkApplication objects that are created.
- A spark serviceaccount exists, which has elevated privileges within the namespace.
- Full access to manage pods, services, configmaps, secrets, and persistent volume claims
- We can permit serviceaccounts in other namespaces to create pods etc.
- Typically, this will be an airflow serviceaccount in an airflow instance namespace.
- Firewall rules are in place, permitting spark pods access to Kerberos, Hive, HDFS, and other services, as required.
- A kerberos keytab has been deployed as a secret, which can be used to authenticate to Hive and HDFS.
We currently have the analytics-test namespace fully enabled in this way and it would be helpful for end users to start using spark and providing feedback to help guide the next steps on configuration.
In order to gain access to the namespace, users will want to run the following from a deployment server.
kube-env analytics-test-deploy dse-k8s-eqiad
These credentials are only accessible to members of the analytics-deployers group, which is effectively an alias for the analytics-admins group. See this page for more information on the shell groups.
The simplest way to get started testing is to make use of the spark-toolbox deployment that is defined within the `spark-support chart.
btullis@deploy2002:~$ kubectl get deployments spark-toolbox NAME READY UP-TO-DATE AVAILABLE AGE spark-toolbox 3/3 3 3 42h
This is a container that uses the spark image and has all of the correct config files, secrets, environment variables etc. but is just running a sleep infinity command, ready for a user to start a shell and run interactive commands.
We can launch a shell in this toolbox like this:
kubectl exec -it spark-toolbox-6fcf874549-n4dff -- /entrypoint.sh bash
The first suffix for the pod will change on each deployment, and the second will be unique to each of the toolbox replicas.
Once inside the toolbox pod, we can start to launch spark sessions in both client and cluster mode.
Client mode sessions include the use of: spark-sql, spark-shell, and pyspark, as well as spark-submit.
Cluster mode sessions just use spark-submit.
When running a client mode setting, there is only one required command-line option, which is spark.driver.host.
In our case, we want to set this value to either the IPv4 or the IPv6 address of the pod.
So the following should all work for starting a spark cluster in client mode.
spark-sql --conf spark.driver.host=[$(hostname -i|awk '{print $1}')]
spark-shell --conf spark.driver.host=[$(hostname -i|awk '{print $1}')]
pyspark --conf spark.driver.host=[$(hostname -i|awk '{print $1}')]In cluster mode, we do not have to override this value.
We have seen the following sparkPi job succeed from a toolbox pod.
spark-submit --verbose --deploy-mode cluster --class org.apache.spark.examples.SparkPi --name "Spark Pi" local:///opt/spark/examples/jars/spark-examples_2.12-3.5.7.jar
The executor pods are created and deleted dynamically, but the driver pod remains in a completed state, so we can check the logs.
btullis@deploy2002:~$ kubectl logs spark-pi-a6237b9b2bcbf64a-driver | tail -n 1 Pi is roughly 3.1441357206786034
There are certain things that we will want to be aware of and to carry on configuring, including:
- Use of kubernetes volumes, including:
- Use of local storage on the dse-k8s-worker nodes.
- Dependency management using HDFS and/or S3 sources.
- Accessing the driver UI.
- Dual-stack networking options.
If we find that there are additional options that we wish to add to the spark or hadoop config files, these will be made to the spark-support chart and the values.yaml file specific to each namespace.
Any feedback on the functionality and testing is welcome.