[Iceberg Migration] Apache Iceberg Migration
Open, Needs TriagePublic
Actions

Assigned To

None

Authored By

	• lbowmaker
	Mar 24 2023, 3:59 PM

Description

User Story

As a Data Engineer I need to migrate datasets to Apache Iceberg

Why?

So that we can improve the management of existing data files, improve query performance and facilitate faster production of incremental datasets

Migration status: https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Iceberg#Which_tables_are_migrated

Related Objects
Search...

Status	Subtype	Assigned	Task
Open		None	T333013 [Iceberg Migration] Apache Iceberg Migration
Declined		None	T333015 Automate installation of Iceberg on the Analytics Cluster
Declined		None	T334601 [needs updating] Migration Dataset: XXX to Apache Iceberg
Resolved		xcollazo	T335305 Migrate referrer_daily to Iceberg
Resolved	Spike	mforns	T335306 [SPIKE] Evaluation on iceberg sensor for airflow
Declined		None	T340463 [Iceberg Migration] P.O.C. on Iceberg sensor using Iceberg table to keep status of updates
Declined		None	T340466 [Iceberg Migration] P.O.C. on Iceberg sensor using Postgres table to keep status of updates
Declined		None	T340471 [Iceberg Migration] P.O.C. on Iceberg sensor using Snapshot metadata to keep status of updates
Resolved		xcollazo	T311738 [Iceberg] Debianize and install iceberg support for Spark, Presto, and optionally Hive
Resolved		Ottomata	T311525 Upgrade to latest PrestoDB and enable iceberg support
Declined		• Stevemunene	T324011 SPIKE: Spin up a Test Trino instance (Evaluate Trino)
Resolved		xcollazo	T335721 Add support for Iceberg in Spark
Open		None	T336012 Add support for Iceberg to the Spark Docker Image
Open		None	T336513 Figure out a way to automatize deployment of the spark assembly file
Resolved		BTullis	T337335 Upgrade Presto to release that aligns with Iceberg 1.2.1
Resolved		BTullis	T295589 Upgrade Presto to access UDF library improvements
Resolved		xcollazo	T335314 Make sure new iceberg data can be queried from Presto and Spark as well as non-Iceberg data
Resolved		xcollazo	T337562 Decide how to split wmf database into functional areas
Open		None	T337416 [Iceberg] performance test to align row group to HDFS block size
Open		None	T338050 Benchmark Iceberg tables with SNAPPY vs GZIP vs ZSTD
Open		None	T338057 Upgrade Spark to a version with long term Iceberg support, and with fixes to support Dumps 2.0
Open		None	T354735 Validating that current jobs are running correctly with the new Conda Analytics environment
Open		None	T354737 Deploy new Spark version on production
Open		None	T378899 Upgrade to Spark 3.2 to support Spark lineage for Iceberg tables
Open		None	T370713 Upgrade to Pyspark ≥ 3.5
Open		None	T370712 Upgrade to Pyspark ≥ 3.4
Open		None	T380030 Create Airflow debian package with pyspark version 3.5.3
Open		brouberol	T380033 Create Airflow docker images with pyspark version 3.5.3
Resolved		BTullis	T380035 Create Spark docker images for version 3.5.3
Resolved		brouberol	T380038 Create a debian package of the spark shuffler for yarn version 3.5.3
Open		None	T380040 Deploy the spark shuffler version 3.5.3 to the test cluster
Open		None	T380041 Deploy the spark shuffler version 3.5.3 to the production cluster
Open		None	T354733 Create Conda Analytics environment including spark version 3.5.3
Resolved		brouberol	T380039 Create and distribute as assembly file for spark version 3.5.3
Open		None	T380398 Update airflow-dags so that spark 3.5.3 shuffler is used by default, with an override to use the v 3.1.2 shuffler if required
Open		None	T380417 Test if an existing conda environment with Spark 3.1.2 clients works fine with Spark 3.5.3
Resolved		xcollazo	T338065 [Iceberg Migration] Implement mechanism for automatic Iceberg table maintenance
Resolved		JAllemandou	T347689 [Iceberg Migration] Migrate unique devices tables to Iceberg
Resolved		nshahquinn-wmf	T349132 Update unique devices table in Movement Insights-owned code
Open		None	T347690 [Iceberg Migration] Migrate pageview tables to Iceberg
Resolved		JAllemandou	T347693 [Iceberg Migration] Deprecate referrer_daily Hive tables
Resolved		Antoine_Quhen	T347879 [Airflow Migration] Migrate Airflow Druid Jobs to Unique Devices Iceberg tables
Resolved		tchin	T352669 [Iceberg Migration] Migrate aqs hourly tables to Iceberg
Resolved		Snwachukwu	T352670 [Iceberg Migration] Migrate browser_general tables to Iceberg
Resolved		tchin	T352671 [Iceberg Migration] Migrate interlanguage tables to Iceberg
Resolved		Antoine_Quhen	T352672 [Iceberg Migration] Migrate session length tables to Iceberg
Resolved	Spike	amastilovic	T360922 [Status Store] [SPIKE] Investigate and document approach for Iceberg Sensors
Resolved		amastilovic	T369900 Develop Airflow ExternalTaskSensor to orchestrate DAG dependencies
Resolved		amastilovic	T372014 Problem deploying - missing airflow_client dependency
Open		amastilovic	T372644 Airflow RestExternalTaskSensor should be able to sense named dynamic mapped tasks
Resolved		amastilovic	T372647 Implement Airflow Dataset class for RestExternalTaskSensor
Resolved		amastilovic	T383931 Unify Airflow's datasets.yaml config files across instances
Resolved		amastilovic	T381432 Create and test a new produced_by config for the datalake table
Resolved		amastilovic	T378000 Write documentation on usage of RestExternalTaskSensor
Open		None	T361503 [Spike] [Maintenance] Define late arrival event strategy and idem-potent backfilling concept.
Open		Snwachukwu	T373693 [Iceberg Migration] Extend Iceberg table maintenance mechanism to support multiple Airflow instances
Open		None	T391135 Iceberg table maintenance for tables under wmf_product database
Resolved		xcollazo	T373694 [Iceberg Migration] Extend Iceberg table maintenance mechanism to support data rewrite