On T344910, we deployed additional Spark Shufflers to our cluster so that we can support Spark 3.3 and Spark 3.4 lines.
We currently use Spark 3.3 on the Dumps 2.0 pipelines.
In this task we should update the jobs so that they leverage the Spark 3.3 shuffler.
| Title | Reference | Author | Source Branch | Dest Branch | |
|---|---|---|---|---|---|
| Use Spark 3.3 Shuffler for Dumps 2.0 pipelines. | repos/data-engineering/airflow-dags!556 | xcollazo | T352890-use-spark-33-shuffler | main |
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Invalid | VirginiaPoundstone | T345988 [Epic] XML MediaWiki data dumps for right to fork | |||
| Resolved | Milimetric | T330296 Dumps 2.0 Phase I: Proof of concept for MediaWiki XML content dump via Event Platform, Iceberg and Spark | |||
| Resolved | xcollazo | T352890 Modify pipelines to leverage Spark 3.3 Shuffler |
xcollazo updated https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/556
Draft: Use Spark 3.3 Shuffler for Dumps 2.0 pipelines.
milimetric merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/556
Use Spark 3.3 Shuffler for Dumps 2.0 pipelines.
Mentioned in SAL (#wikimedia-analytics) [2023-12-07T21:45:30Z] <xcollazo> Deployed latest changes to Airflow Analytics instance to pickup T352890