We are currently running Spark 3.1.2.
Since 3.1.3, the connection with the legacy shuffle service has been broken.
We should find a way to upgrade the last Spark version by:

• making the last version works with the legacy shuffle service (from Spark 2)
• or switching to the new internal way of managing shuffle files with the driver + permanent external storage.

It impacts the spark autoscaling.

Note: the legacy shuffle service, currently supported by Yarn, is not supported by K8S (and probably won't be ported).