Modify pipelines to leverage Spark 3.3 Shuffler
Closed, ResolvedPublic2 Estimated Story Points
Actions

Assigned To

Authored By

	xcollazo
	Dec 6 2023, 4:53 PM

Description

On T344910, we deployed additional Spark Shufflers to our cluster so that we can support Spark 3.3 and Spark 3.4 lines.

We currently use Spark 3.3 on the Dumps 2.0 pipelines.

In this task we should update the jobs so that they leverage the Spark 3.3 shuffler.

		Status	Subtype	Assigned	Task
		Resolved		Milimetric	T330296 Dumps 2.0 Phase I: Proof of concept for MediaWiki XML content dump via Event Platform, Iceberg and Spark
		Resolved		xcollazo	T352890 Modify pipelines to leverage Spark 3.3 Shuffler

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptDec 6 2023, 4:53 PM

xcollazo set the point value for this task to 2.Dec 6 2023, 4:54 PM

Draft: Use Spark 3.3 Shuffler for Dumps 2.0 pipelines.

Use Spark 3.3 Shuffler for Dumps 2.0 pipelines.

Mentioned in SAL (#wikimedia-analytics) [2023-12-07T21:45:30Z] <xcollazo> Deployed latest changes to Airflow Analytics instance to pickup T352890

Deployed to production, working as expected.

xcollazo closed this task as Resolved.Jan 2 2024, 5:18 PM