Dumps 2.0 Phase III: Production level dumps
Open, Needs TriagePublic
Actions

Assigned To

None

Authored By

	xcollazo
	Jun 5 2024, 8:18 PM

Description

(Previous phase: T358877: Dumps 2.0 Phase II: Production intermediate table milestone)

Achieving this milestone would mean that we are in a position to consider decommissioning the XML Dumps from Dumps 1.0 in favor of Dumps 2.0.

Related Objects
Search...

Status	Assigned	Task
Open	None	T366752 Dumps 2.0 Phase III: Production level dumps
Resolved	BTullis	T374952 Figure why we can't dump labswiki, aka Wikitech
Open	None	T363941 dumps_publish_wikitext_raw_to_xml DAG fails sporadically. Needs sensors to wait before running.
Resolved	BTullis	T371720 Bump Presto to latest pickup Iceberg fixes and VIEWs support
Open	BTullis	T338057 Upgrade Spark to a version with long term Iceberg support, and with fixes to support Dumps 2.0
Open	None	T354735 Validating that current jobs are running correctly with the new Conda Analytics environment
Open	None	T354737 Deploy new Spark version on production
Open	None	T378899 Upgrade to Spark 3.2 to support Spark lineage for Iceberg tables
Open	None	T370713 Upgrade to Pyspark ≥ 3.5
Open	None	T370712 Upgrade to Pyspark ≥ 3.4
Open	brouberol	T380030 Create Airflow debian package with pyspark version 3.5.3
Open	brouberol	T380033 Create Airflow docker images with pyspark version 3.5.3
Resolved	BTullis	T380035 Create Spark docker images for version 3.5.3
Resolved	brouberol	T380038 Create a debian package of the spark shuffler for yarn version 3.5.3
Open	BTullis	T380040 Deploy the spark shuffler version 3.5.3 to the test cluster
Open	None	T380041 Deploy the spark shuffler version 3.5.3 to the production cluster
Open	BTullis	T354733 Create Conda Analytics environment including spark version 3.5.3
Resolved	brouberol	T380039 Create and distribute as assembly file for spark version 3.5.3
Open	BTullis	T380398 Update airflow-dags so that spark 3.5.3 shuffler is used by default, with an override to use the v 3.1.2 shuffler if required
Open	xcollazo	T380417 Test if an existing conda environment with Spark 3.1.2 clients works fine with Spark 3.5.3
Open	None	T379365 [SPIKE] Benchmark the run time of batch processing
Open	None	T379410 Go over tasks in #dumps-generation and figure what makes sense to fix in Dumps 2.0
Open	None	T379921 Implement a canary against MariaDB schema changes
Open	None	T366544 Use the Spark-Iceberg built in CDC mechanism to PoC a replacement for wikimedia_wikitext_current
Open	None	T380773 [Dumps 2] Time partitioning for mediawiki_content_history
Open	None	T379676 Add relevant kafka clusters to defined airflow connections in puppet
In Progress	pfischer	T381016 Figure why XML Dump code generates 1000's of files for simplewiki
In Progress	Milimetric	T381615 mediawiki-jdbc, our custom spark data source, doesn't fully hit the indexes
Open	None	T371509 Consider whether we want to dump private wikis