Change Details

On T344910, we deployed additional Spark Shufflers to our cluster so that we can support Spark 3.3 and Spark 3.4 lines. We currently use Spark 3.3 on the Dumps 2.0 pipelines. In this task we should update the jobs so that they leverage the Spark 3.3 shuffler. [] All Dump 2.0 pipelines run with the Spark Shuffler.

On T344910, we deployed additional Spark Shufflers to our cluster so that we can support Spark 3.3 and Spark 3.4 lines. We currently use Spark 3.3 on the Dumps 2.0 pipelines. In this task we should update the jobs so that they leverage the Spark 3.3 shuffler. [x] All Dump 2.0 pipelines run with the Spark Shuffler.

On T344910, we deployed additional Spark Shufflers to our cluster so that we can support Spark 3.3 and Spark 3.4 lines. We currently use Spark 3.3 on the Dumps 2.0 pipelines. In this task we should update the jobs so that they leverage the Spark 3.3 shuffler. [x] All Dump 2.0 pipelines run with the Spark Shuffler.