Provide a Spark-on-k8s access for sql tools (dbt)
Open, HighPublic
Actions

Assigned To

None

Authored By

	JAllemandou
	Nov 13 2025, 10:28 AM

Description

Update January 2026

Our work is currently focused on: T413977: Deploy Kyuubi to enable dynamic spark-sql clusters in dse-k8s

We had hoped to use spark-thriftserver in a Kubernetes sidecar model.
This would have allowed the creation of a custom, ephemeral spark cluster, alongside each dbt job launched by airflow.

Achieving this has proved difficult because of the fact that our Hive Metastore and HDFS cluster are protected by Kerberos.
Specifically, we cannot run the spark thriftserver and its JDBC interface without also having it be kerberos-enabled.
This kerberos-enabled service is broadly incompatible with ephemeral services in short-lived kubernetes pods, due to the way in which kerberos wants to resolve host names.

For this reason, we are now moving towards having a cetral Kyuubi service, per analytics namespace.
This will still allow us to create ephemeral spark clusters with thrift and JDBC interfaces, but the hostnames and pod names will be static and therefore we will be able to make kerberos work using our current keytab deployment mechanism.

Original description follows

For testing dbt jobs, spark session-mode starts spark jobs from stat machines.

For production jobs launched via airflow, getting the session-mode will be more complicated as it would mean running the dbt from within a skein container.

One different approach would be to provide one SparkThriftSQLServer per production user, allowing Airflow instances to run dbt jobs against those Servers.

If using skein, the idea would be to test launching dbt from a skein container, with a specific profile for spark to run in session mode.

If using Spark Thrift-Server the idea would be to
- install one SparkThriftServer using the analytics user
- in k8s if possible, on bare metal/VM otherwise,
- With auto-restart in case of failure (k8s would help for this) as this would use regularly from Airflow
- with a defined DNS to be accessed from Airflow
- Verify that the job can execute SQL and write production data

If we can have all that, the next steps will be to make the replication of such a server easy, to provide this capability for every prod-user on the Hadoop cluster.

Details

Related Changes in Gerrit:

Subject	Repo	Branch	Lines +/-
Bind the spark-job-orchestration role to the default serviceaccount	operations/deployment-charts	master	+3 -0
Update the image used for the spark-toolbox	operations/deployment-charts	master	+1 -1
Fix the name of the kerberos-client-configuration configmap	operations/deployment-charts	master	+2 -2
Update the configmap names for spark-support	operations/deployment-charts	master	+10 -10
Update the volumes and volumemounts of the kyuubi toolbox	operations/deployment-charts	master	+129 -100
Configure the contents of /etc/kyuubi/conf for the kyuubi toolbox pod	operations/deployment-charts	master	+33 -13
Add a kyuubi deployment to the spark-support chart for analytics-test	operations/deployment-charts	master	+60 -12
Revert changes to the principal for the spark-thriftserver	operations/deployment-charts	master	+2 -2
Update the spark config	operations/deployment-charts	master	+8 -5
Update the hostname used by dbt in the spark-support chart	operations/deployment-charts	master	+2 -3
Add a dbt profiles.yml file to the spark-support chart	operations/deployment-charts	master	+32 -2
Allow the spark serviceaccount to perform more actions	operations/deployment-charts	master	+4 -4
Fix the keytab path for spark-support	operations/deployment-charts	master	+1 -1
Update the kerberos settings for the hive thriftserver	operations/deployment-charts	master	+5 -1
Update the spark image that is deployed to analytics-test	operations/deployment-charts	master	+1 -1
Update the spark-support chart	operations/deployment-charts	master	+0 -19
Bump spark image	operations/deployment-charts	master	+1 -1
Update the spark configuration	operations/deployment-charts	master	+29 -36
Remove unnecessary and/or incorrect hadoop/spark config options	operations/deployment-charts	master	+1 -67
Add k8s tokens for the analytics namespace	operations/puppet	production	+8 -0
Add an analytics namespace to both dse-k8s clusters	operations/deployment-charts	master	+4 -0
Enable the deploy-spark-support deploy clusterrole for two test namespaces	operations/deployment-charts	master	+2 -0
Add a new deploy-spark-support clusterrole	operations/deployment-charts	master	+40 -0

Related Changes in GitLab:

Title	Reference	Author	Source Branch	Dest Branch
Update the kyuubi variants	repos/data-engineering/spark!50	btullis	add_kyuubi	main
Add a variant for kyuubi	repos/data-engineering/spark!48	btullis	add_kyuubi	main
Add the pykerberos package to the dbt venv	repos/data-engineering/dbt-jobs!14	btullis	add_python_kerberos	main
Add support for spark in thrift mode	repos/data-engineering/dbt-jobs!13	btullis	add_pyhive	main
Add Kerberos packages to the docker image	repos/data-engineering/dbt-jobs!12	javiermonton	feature/add-kerberos-packages	main
Add kerberos user utilities to the spark distribution	repos/data-engineering/spark!47	btullis	add_krb5_spark	main
Update the JDK/JRE to Java 11	repos/data-engineering/spark!45	btullis	update_jre	main
Add the libisal2 and libssl-dev packages	repos/data-engineering/spark!44	btullis	add_missing_runtime_libs	main
Rework the blubber file to reduce repetition and add the iceberg jar	repos/data-engineering/spark!43	btullis	add_iceberg_support	main
Upgrade the Hadoop distribution to 3.4.2 when building spark images	repos/data-engineering/spark!42	btullis	update_hadoop_version	main
Add snappy and ssl support to spark images	repos/data-engineering/spark!39	btullis	add_snappy_ssl_support	main
Use the lowercase env variables for proxy settings	repos/data-engineering/spark!26	btullis	fix_mvn_proxy_opts	main
Use a different method for setting the maven proxy configuration	repos/data-engineering/spark!25	btullis	fix_maven_proxy	main
Update the maven opts for the publish stage	repos/data-engineering/spark!24	btullis	fix_maven_opt	main
Set the MAVEN_OPTS variable when building on trusted runners	repos/data-engineering/spark!23	btullis	fix_maven_proxy_opts	main
Build a custom spark distribution with hive and thriftserver support	repos/data-engineering/spark!22	btullis	build_spark	main
Add python3-dev and procps to the spark distribution image	repos/data-engineering/spark!21	btullis	update_spark_image	main
Create a dbt DAG that launches a pod containing a dbt-jobs pod and a spark-thriftserver sidecar	repos/data-engineering/airflow-dags!1820	btullis	spark_thrift_dbt	main

Related Objects
Search...

Status	Assigned	Task
Open	Mayakp.wiki	T369226 [MI 1] Reporting, Data Visualization, and Data Communications
Resolved	Mayakp.wiki	T403304 [MI 1.2] Provide metric updates to staff through 2 metric-focused presentations
Resolved	Mayakp.wiki	T403305 Contributors Metrics Reporting for Staff and stakeholders
Resolved	Mayakp.wiki	T408146 [Hypothesis] WE1.5.1 Contributor metrics dashboard
Open	JMonton-WMF	T416679 dbt DPE work
Open	amastilovic	T406764 Provide a dbt-core development environment and production setup in the data-platform
Open	None	T410017 Provide a Spark-on-k8s access for sql tools (dbt)
Open	None	T413977 Deploy Kyuubi to enable dynamic spark-sql clusters in dse-k8s

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

btullis merged https://gitlab.wikimedia.org/repos/data-engineering/spark/-/merge_requests/21

Add python3-dev and procps to the spark distribution image

Change #1212130 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/deployment-charts@master] Bind the spark-job-orchestration role to the default serviceaccount

https://gerrit.wikimedia.org/r/1212130

btullis opened https://gitlab.wikimedia.org/repos/data-engineering/spark/-/merge_requests/22

Build a custom spark distribution with hive and thriftserver support

btullis merged https://gitlab.wikimedia.org/repos/data-engineering/spark/-/merge_requests/22

Build a custom spark distribution with hive and thriftserver support

btullis opened https://gitlab.wikimedia.org/repos/data-engineering/spark/-/merge_requests/23

Set the MAVEN_OPTS variable when building on trusted runners

btullis merged https://gitlab.wikimedia.org/repos/data-engineering/spark/-/merge_requests/23

Set the MAVEN_OPTS variable when building on trusted runners

btullis opened https://gitlab.wikimedia.org/repos/data-engineering/spark/-/merge_requests/24

Update the maven opts for the publish stage

btullis merged https://gitlab.wikimedia.org/repos/data-engineering/spark/-/merge_requests/24

Update the maven opts for the publish stage

btullis opened https://gitlab.wikimedia.org/repos/data-engineering/spark/-/merge_requests/25

Use a different method for setting the maven proxy configuration

btullis merged https://gitlab.wikimedia.org/repos/data-engineering/spark/-/merge_requests/25

Use a different method for setting the maven proxy configuration

btullis opened https://gitlab.wikimedia.org/repos/data-engineering/spark/-/merge_requests/26

Use the lowercase env variables for proxy settings

btullis merged https://gitlab.wikimedia.org/repos/data-engineering/spark/-/merge_requests/26

Use the lowercase env variables for proxy settings

I ran into a bit of a problem because the pre-built versions of spark did not include support for hive or the thriftserver.

btullis@deploy2002:~$ kubectl exec -it dbt-debug-ry7aiyd -c spark-thriftserver -- /entrypoint.sh bash
<snip>
spark@dbt-debug-ry7aiyd:/tmp$ /opt/spark/sbin/start-thriftserver.sh
<snip>
  ========================================
  Error: Failed to load class org.apache.spark.sql.hive.thriftserver.HiveThriftServer2.
  Failed to load main class org.apache.spark.sql.hive.thriftserver.HiveThriftServer2.
  You need to build Spark with -Phive and -Phive-thriftserver.

So in order to get:

version 3.5.7 of spark
alongside version 2.10.2 of hadoop
with the correct build flags to support the thriftserver

... I needed to update the build process to build spark from source.

I've done that now, so the latest version of the spark image now has support for running the thriftserver:

The next step is to get the default serviceaccount in the analytics-test namespace the required privileges to create and manage pods etc.

I've been running this command as a test:

spark-submit --master k8s://https://kubernetes.default \
 --deploy-mode cluster \
 --name spark-pi \
 --class org.apache.spark.examples.SparkPi \
 --conf spark.executor.instances=5 \
 --conf spark.kubernetes.container.image=docker-registry.discovery.wmnet/repos/data-engineering/spark:3.5.7-2025-12-01-152105-a7a1e7a2edebcf5343245f56a7efab1be4b317b4 \
 --conf spark.kubernetes.namespace=analytics-test \
 local:///opt/spark/examples/jars/spark-examples_2.12-3.5.7.jar

The error messages indicate that the default serviceaccount cannot create pods.

16:34:53.098 [main] ERROR org.apache.spark.deploy.k8s.submit.Client - Please check "kubectl auth can-i create pod" first. It should be yes.
Exception in thread "main" io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: POST at: https://kubernetes.default/api/v1/namespaces/analytics-test/pods. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. pods is forbidden: User "system:serviceaccount:analytics-test:default" cannot create resource "pods" in API group "" in the namespace "analytics-test".

We could also look at using a custom serviceaccount. e.g.

--conf spark.kubernetes.authenticate.driver.serviceAccountName=spark

...but in my testing, this didn't seem to be applied correctly, and I was led to this bug: https://issues.apache.org/jira/browse/SPARK-26295

Therefore, it might be better to modify the default serviceaccount for each spark-enabled namespace.

btullis opened https://gitlab.wikimedia.org/repos/data-engineering/spark/-/merge_requests/39

Add snappy and ssl support to spark images

Change #1214020 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/deployment-charts@master] Remove unnecessary and/or incorrect hadoop/spark config options

https://gerrit.wikimedia.org/r/1214020

btullis merged https://gitlab.wikimedia.org/repos/data-engineering/spark/-/merge_requests/39

Add snappy and ssl support to spark images

Change #1214020 merged by jenkins-bot:

[operations/deployment-charts@master] Remove unnecessary and/or incorrect hadoop/spark config options

https://gerrit.wikimedia.org/r/1214020

JAllemandou moved this task from In progress to Blocked/Paused on the Data-Engineering (Q2 FY25/26 October 1st - December 31th) board.Dec 2 2025, 1:11 PM

Change #1214091 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/deployment-charts@master] Update the spark configuration

https://gerrit.wikimedia.org/r/1214091

Change #1214091 merged by jenkins-bot:

[operations/deployment-charts@master] Update the spark configuration

https://gerrit.wikimedia.org/r/1214091

bking subscribed.Dec 2 2025, 4:28 PM

I'm still making steady progress on this, but there is one more thing that I might have to fix, which is the openssl support.

I'm getting the following output from hadoop checknative in the spark container.

spark@dbt-debug-m5phhkk:/tmp$ hadoop checknative
25/12/02 19:02:58 INFO bzip2.Bzip2Factory: Successfully loaded & initialized native-bzip2 library system-native
25/12/02 19:02:58 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
Native library checking:
hadoop:  true /opt/hadoop/lib/native/libhadoop.so.1.0.0
zlib:    true /lib/x86_64-linux-gnu/libz.so.1
snappy:  true /lib/x86_64-linux-gnu/libsnappy.so.1
zstd  :  true /lib/x86_64-linux-gnu/libzstd.so.1
lz4:     true revision:10301
bzip2:   true /lib/x86_64-linux-gnu/libbz2.so.1
openssl: false EVP_CIPHER_CTX_block_size

This led me to these bugs:

Ultimately, I believe that this is related to the fact that Debian bookworm uses openssl version 3 and Hadoop 2.10.2 still needs libssl1.1

I think that I can get around this by using the same workaround that I did here.

btullis opened https://gitlab.wikimedia.org/repos/data-engineering/spark/-/merge_requests/42

Upgrade the Hadoop distribution to 3.3.6 when building spark images

I'm taking a slightly different approach to this, since I discovered a problem using Spark 3.5.7 with Hadoop 2.10.2.

I had got to the point where I was running the following command in a spark container:

pyspark --conf spark.kerberos.keytab=/etc/security/keytabs/analytics.keytab --conf spark.kerberos.principal=analytics/analytics-test.discovery.wmnet@WIKIMEDIA

The spark-defaults.conf file at this point contains the following configuration statements.

spark@dbt-debug-m5phhkk:/tmp$ cat /opt/spark/conf/spark-defaults.conf

spark.authenticate  true
spark.dynamicAllocation.cachedExecutorIdleTimeout  3600s
spark.dynamicAllocation.enabled  true
spark.dynamicAllocation.executorIdleTimeout  60s
spark.eventLog.compress  true
spark.eventLog.dir  hdfs:///var/log/spark
spark.eventLog.enabled  true
spark.kubernetes.authenticate.driver.serviceAccountName  spark
spark.network.crypto.enabled  true
spark.network.crypto.keyFactoryAlgorithm  PBKDF2WithHmacSHA256
spark.network.crypto.keyLength  256
spark.network.crypto.saslFallback  false
spark.sql.catalog.spark_catalog  org.apache.iceberg.spark.SparkSessionCatalog
spark.sql.catalog.spark_catalog.type  hive
spark.sql.catalogImplementation  hive
spark.sql.extensions  org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions
spark.sql.files.maxPartitionBytes  268435456
spark.sql.warehouse.dir  hdfs:///user/hive/warehouse

The following error message appeared.

py4j.protocol.Py4JJavaError: An error occurred while calling None.org.apache.spark.api.java.JavaSparkContext.
: java.lang.NoSuchMethodError: org.apache.hadoop.hdfs.DistributedFileSystem$HdfsDataOutputStreamBuilder.replicate()Lorg/apache/hadoop/hdfs/DistributedFileSystem$HdfsDataOutputStreamBuilder;
	at org.apache.spark.deploy.SparkHadoopUtil$.createFile(SparkHadoopUtil.scala:578)
	at org.apache.spark.deploy.history.EventLogFileWriter.initLogFile(EventLogFileWriters.scala:98)
	at org.apache.spark.deploy.history.SingleEventLogFileWriter.start(EventLogFileWriters.scala:223)
	at org.apache.spark.scheduler.EventLoggingListener.start(EventLoggingListener.scala:81)
	at org.apache.spark.SparkContext.<init>(SparkContext.scala:632)
	at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:58)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:247)
	at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:374)
	at py4j.Gateway.invoke(Gateway.java:238)
	at py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:80)
	at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:69)
	at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182)
	at py4j.ClientServerConnection.run(ClientServerConnection.java:106)
	at java.lang.Thread.run(Thread.java:750)

I believe that this indicates a serious impediment to continuing to use Hadoop 2.10.2 with Spark 3.5.7 because this method:

DistributedFileSystem$HdfsDataOutputStreamBuilder.replicate()

...was only introduced in Hadoop 3.3.

At this point, I believe that we have two options to continue using Spark on Kubernetes for this project:

Upgrade Hadoop and rely on the backward compatibility of the HDFS version 3.3 jars
Downgrade Spark to a version that we know works with the HDFS version 2.10 jars

I'm investigating option 1) for this, first. It does mean that we have to drop support for YARN from this spark build, because we know that the YARN client jars are not backward compatible.
However, I don't believe that this poses any difficulty for using our existing hive metastore, since that uses a Thrift API, which should be stable between these versions.

btullis merged https://gitlab.wikimedia.org/repos/data-engineering/spark/-/merge_requests/42

Upgrade the Hadoop distribution to 3.4.2 when building spark images

btullis opened https://gitlab.wikimedia.org/repos/data-engineering/spark/-/merge_requests/43

Rework the blubber file to reduce repetition and add the iceberg jar

btullis merged https://gitlab.wikimedia.org/repos/data-engineering/spark/-/merge_requests/43

Rework the blubber file to reduce repetition and add the iceberg jar

btullis opened https://gitlab.wikimedia.org/repos/data-engineering/spark/-/merge_requests/44

Add the libisal2 and libssl-dev packages

btullis merged https://gitlab.wikimedia.org/repos/data-engineering/spark/-/merge_requests/44

Add the libisal2 and libssl-dev packages

Change #1215556 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/deployment-charts@master] Bump spark image

https://gerrit.wikimedia.org/r/1215556

I am continuing to make progress, but now I have an issue with the iceberg version:

spark@dbt-debug-kjlbkxo:/tmp$ spark-sql
Exception in thread "main" java.lang.UnsupportedClassVersionError: org/apache/iceberg/spark/ExtendedParser has been compiled by a more recent version of the Java Runtime (class file version 55.0), this version of the Java Runtime only recognizes class file versions up to 52.0

I have been trying to use Iceberg version 1.10.0

The last version of Iceberg to support Java 8 was version 1.6.1 - but that would mean downgrading Spark to version 3.5.1

I will try with a Java 11 based build, instead.

btullis opened https://gitlab.wikimedia.org/repos/data-engineering/spark/-/merge_requests/45

Update the JDK/JRE to Java 11

btullis merged https://gitlab.wikimedia.org/repos/data-engineering/spark/-/merge_requests/45

Update the JDK/JRE to Java 11

Change #1215556 merged by jenkins-bot:

[operations/deployment-charts@master] Bump spark image

https://gerrit.wikimedia.org/r/1215556

btullis opened https://gitlab.wikimedia.org/repos/data-engineering/spark/-/merge_requests/47

Add kerberos user utilities to the spark distribution

btullis merged https://gitlab.wikimedia.org/repos/data-engineering/spark/-/merge_requests/47

Add kerberos user utilities to the spark distribution

javiermonton merged https://gitlab.wikimedia.org/repos/data-engineering/dbt-jobs/-/merge_requests/12

Add Kerberos packages to the docker image

Change #1217495 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/deployment-charts@master] Update the spark-support chart

https://gerrit.wikimedia.org/r/1217495

Change #1217495 merged by jenkins-bot:

[operations/deployment-charts@master] Update the spark-support chart

https://gerrit.wikimedia.org/r/1217495

Change #1217531 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/deployment-charts@master] Update the spark image that is deployed to analytics-test

https://gerrit.wikimedia.org/r/1217531

Change #1217531 merged by jenkins-bot:

[operations/deployment-charts@master] Update the spark image that is deployed to analytics-test

https://gerrit.wikimedia.org/r/1217531

Change #1217535 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/deployment-charts@master] Update the kerberos settings for the hive thriftserver

https://gerrit.wikimedia.org/r/1217535

Change #1217535 merged by jenkins-bot:

[operations/deployment-charts@master] Update the kerberos settings for the hive thriftserver

https://gerrit.wikimedia.org/r/1217535

Change #1217554 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/deployment-charts@master] Fix the keytab path for spark-support

https://gerrit.wikimedia.org/r/1217554

Change #1217554 merged by jenkins-bot:

[operations/deployment-charts@master] Fix the keytab path for spark-support

https://gerrit.wikimedia.org/r/1217554

Change #1217568 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/deployment-charts@master] Allow the spark serviceaccount to perform more actions

https://gerrit.wikimedia.org/r/1217568

Change #1217568 merged by jenkins-bot:

[operations/deployment-charts@master] Allow the spark serviceaccount to perform more actions

https://gerrit.wikimedia.org/r/1217568

Change #1218271 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/deployment-charts@master] Add a dbt profiles.yml file to the spark-support chart

https://gerrit.wikimedia.org/r/1218271

Change #1218271 merged by jenkins-bot:

[operations/deployment-charts@master] Add a dbt profiles.yml file to the spark-support chart

https://gerrit.wikimedia.org/r/1218271

btullis opened https://gitlab.wikimedia.org/repos/data-engineering/dbt-jobs/-/merge_requests/13

Add support for spark in thrift mode

btullis merged https://gitlab.wikimedia.org/repos/data-engineering/dbt-jobs/-/merge_requests/13

Add support for spark in thrift mode

btullis opened https://gitlab.wikimedia.org/repos/data-engineering/dbt-jobs/-/merge_requests/14

Add the pykerberos package to the dbt venv

btullis merged https://gitlab.wikimedia.org/repos/data-engineering/dbt-jobs/-/merge_requests/14

Add the pykerberos package to the dbt venv

Change #1218379 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/deployment-charts@master] Update the hostname used by dbt in the spark-support chart

https://gerrit.wikimedia.org/r/1218379

Change #1218379 merged by jenkins-bot:

[operations/deployment-charts@master] Update the hostname used by dbt in the spark-support chart

https://gerrit.wikimedia.org/r/1218379

Change #1218741 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/deployment-charts@master] Update the spark config

https://gerrit.wikimedia.org/r/1218741

Change #1218741 merged by jenkins-bot:

[operations/deployment-charts@master] Update the spark config

https://gerrit.wikimedia.org/r/1218741

I've run into a hard blocker with the spark-thriftserver sidecar method, so we are currently looking at alternative options.

The reason for the blocker is as follows:

When connecting to an upstream hive-metastore that uses Kerberos, it seems to be required also to use kerberos for the local thrift server port.
The kerberos authentication of the hive thriftserver service is also failing, because it wants the hostname part of the principal to match the reverse DNS lookup of the pod.

I.e. if we configure the principal of the services as: analytics/analytics-test.discovery.wmnet@WIKIMEDIA then the thrift servers process wants the pod's reverse DNS to resolve to analytics-test.discovery.wmnet
If this isn't the case, then the service will not start.

I tried to use some hostAlias entries that caused analytics-test.discovery.wmnet to resolve back to 127.0.0.1 but this didn't work, either.
The kerberos authentication might have been OK, but this DNS trick caused Java to bind only to the localhost interface, so it couldn't reach out to HDFS etc.

So now we're looking into alternative ways of getting this to work.

One of the most promising methods looks to be using Apache Kyuubi in this sidecar pattern.
We will investigate whether we can do this and make use of the connection level isolation.

btullis opened https://gitlab.wikimedia.org/repos/data-engineering/spark/-/merge_requests/48

Add a variant for kyuubi

btullis merged https://gitlab.wikimedia.org/repos/data-engineering/spark/-/merge_requests/48

Add a variant for kyuubi

Change #1220369 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/deployment-charts@master] Revert changes to the principal for the spark-thriftserver

https://gerrit.wikimedia.org/r/1220369

Change #1220369 merged by jenkins-bot:

[operations/deployment-charts@master] Revert changes to the principal for the spark-thriftserver

https://gerrit.wikimedia.org/r/1220369

btullis opened https://gitlab.wikimedia.org/repos/data-engineering/spark/-/merge_requests/50

Update the kyuubi variants

btullis merged https://gitlab.wikimedia.org/repos/data-engineering/spark/-/merge_requests/50

Update the kyuubi variants

Change #1220644 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/deployment-charts@master] Add a kyuubi deployment to the spark-support chart for analytics-test

https://gerrit.wikimedia.org/r/1220644

Change #1220644 merged by jenkins-bot:

[operations/deployment-charts@master] Add a kyuubi deployment to the spark-support chart for analytics-test

https://gerrit.wikimedia.org/r/1220644

Change #1223646 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/deployment-charts@master] Configure the contents of /etc/kyuubi/conf for the kyuubi toolbox pod

https://gerrit.wikimedia.org/r/1223646

Change #1223646 merged by jenkins-bot:

[operations/deployment-charts@master] Configure the contents of /etc/kyuubi/conf for the kyuubi toolbox pod

https://gerrit.wikimedia.org/r/1223646

Change #1223661 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/deployment-charts@master] Update the volumes and volumemounts of the kyuubi toolbox

https://gerrit.wikimedia.org/r/1223661

Change #1223661 merged by jenkins-bot:

[operations/deployment-charts@master] Update the volumes and volumemounts of the kyuubi toolbox

https://gerrit.wikimedia.org/r/1223661

Change #1223663 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/deployment-charts@master] Update the configmap names for spark-support

https://gerrit.wikimedia.org/r/1223663

Change #1223663 merged by jenkins-bot:

[operations/deployment-charts@master] Update the configmap names for spark-support

https://gerrit.wikimedia.org/r/1223663

Change #1223668 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/deployment-charts@master] Fix the name of the kerberos-client-configuration configmap

https://gerrit.wikimedia.org/r/1223668

Change #1223668 merged by jenkins-bot:

[operations/deployment-charts@master] Fix the name of the kerberos-client-configuration configmap

https://gerrit.wikimedia.org/r/1223668

Time for a further update on this.

We have been experimenting with running kyuubi in a toolbox pod, in the same way that we were attempting to run the spark-thriftserver.

The intention had been to use this in an ephemeral manner, in the way that we had previously been attempting to do with the spark-thriftserver.
However, we have run up against a very similar problem with kyuubi in this configuration, to that we faced with the spark-thriftserver.

Namely, when the Hive Metastore and HDFS cluster to which the Kyuubi engine connects are protected by Kerberos, the Kyuubi server itself also needs to run with Kerberos enabled.
We cannot easily do this with ephemeral pods because we do not automatically generate kerberos principals that can match the pod name. Our principals are managed manually and the keytabs are statically deployed to kubernetes, so this does not work well when we wish to have multiple kyuubi servers.

Having discussed it with @JAllemandou, we have come up with an alternative plan. We now intend to change the deployment plan for Kyuubi, so that instead of creating ephemeral kyuubi servers, we will use a single instance per namespace.
This will allow us to use a fixed pod name that will match our kerberos principal FQDN and we will be able to use a kubernetes service to allow clients to connect using the same hostname.

Our DBT containers will need to have their own kerberos keytab and credential renewer, but will no longer be looking at running a sidecar to run kyuubi or spark-thriftserver.

We will still use the CONNECTION share level, so we will still get a spark cluster created per JDBC connection from a DBT job.
This will also allow customization of each spark cluster using the JDBC connection string, such as:

jdbc:hive2://localhost:10009/default;#spark.sql.shuffle.partitions=2;spark.executor.memory=5g

I will create a new ticket for this work to deploy and test a single Kyuubi instance per analytics-enabled namespace.

BTullis mentioned this in T413977: Deploy Kyuubi to enable dynamic spark-sql clusters in dse-k8s.Jan 7 2026, 1:38 PM

BTullis added a subtask: T413977: Deploy Kyuubi to enable dynamic spark-sql clusters in dse-k8s.

BTullis moved this task from In Progress to Blocked/Waiting on the Data-Platform-SRE (2025.11.07 - 2025.11.28) board.

BTullis updated the task description. (Show Details)Jan 7 2026, 1:51 PM

dr0ptp4kt subscribed.Jan 7 2026, 3:51 PM

Gehel edited projects, added Data-Platform-SRE (2026.01.05 - 2026.01.23); removed Data-Platform-SRE (2025.11.07 - 2025.11.28).Jan 8 2026, 1:13 PM

Gehel moved this task from Backlog - project to Blocked/Waiting on the Data-Platform-SRE (2026.01.05 - 2026.01.23) board.

Ahoelzl edited projects, added Data-Engineering (Q3 FY25/26 January 1st - March 31th); removed Data-Engineering (Q2 FY25/26 October 1st - December 31th).Jan 8 2026, 4:32 PM

Ahoelzl moved this task from Next Up to Blocked/Paused on the Data-Engineering (Q3 FY25/26 January 1st - March 31th) board.

is this task paused because its dependent on the success of T413977 ?

Change #1226261 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/deployment-charts@master] Update the image used for the spark-toolbox

https://gerrit.wikimedia.org/r/1226261

Change #1226261 merged by jenkins-bot:

[operations/deployment-charts@master] Update the image used for the spark-toolbox

https://gerrit.wikimedia.org/r/1226261

Change #1212130 abandoned by Btullis:

[operations/deployment-charts@master] Bind the spark-job-orchestration role to the default serviceaccount

Reason:

No longer needed

https://gerrit.wikimedia.org/r/1212130

Gehel edited projects, added Data-Platform-SRE (2026.01.23 - 2026.02.13); removed Data-Platform-SRE (2026.01.05 - 2026.01.23).Jan 23 2026, 10:02 AM

Gehel moved this task from Backlog - project to Blocked/Waiting on the Data-Platform-SRE (2026.01.23 - 2026.02.13) board.

JAllemandou renamed this task from Provide a Spark production access for dbt with Airflow to Provide a Spark-on-k8s access for sql tools (dbt).Feb 4 2026, 11:24 AM

JAllemandou moved this task from Blocked/Waiting to Backlog - project on the Data-Platform-SRE (2026.01.23 - 2026.02.13) board.

Gehel edited projects, added Data-Platform-SRE (2026-02-13 - 2026-03-06); removed Data-Platform-SRE (2026.01.23 - 2026.02.13).Mon, Feb 16, 3:36 PM

I'm moving this back to the main Data-Platform-SRE backlog, since we're not working on it in the immediate future.
We have implemented a solution for dbt by adding it to conda-analytics and launching this on YARN, via skein.

That is good enough for now, but we still want to have a way to do this nicely with spark running natively under kuibernetes.
Therefore, I will leave it open and unassign myself.

BTullis moved this task from Incoming to Infrastructure on the Data-Platform-SRE board.Tue, Feb 24, 2:24 PM

Provide a Spark-on-k8s access for sql tools (dbt)Open, HighPublicActions