⚓ T373693 [Iceberg Migration] Extend Iceberg table maintenance mechanism to support multiple Airflow instances

	Title	Reference	Author	Source Branch	Dest Branch
	Add Iceberg maintenance DAG and Spark 3.3.2 defaults for all instances having iceberg datasets	repos/data-engineering/airflow-dags!1992	ebysans	wmde_table_maintenance	main
	Support Multi-Instance Iceberg table maintenance	repos/data-engineering/airflow-dags!1977	ebysans	multi_instance_maintenance_iceberg	main

Status	Assigned	Task
Open	None	T333013 [Iceberg Migration] Apache Iceberg Migration
Resolved	Snwachukwu	T373693 [Iceberg Migration] Extend Iceberg table maintenance mechanism to support multiple Airflow instances
Resolved	KCVelaga_WMF	T391135 Iceberg table maintenance for tables under wmf_product database

xcollazo created this task.Aug 30 2024, 2:14 PM

Restricted Application added a subscriber: • Aklapper. · View Herald TranscriptAug 30 2024, 2:14 PM

xcollazo mentioned this in T338065: [Iceberg Migration] Implement mechanism for automatic Iceberg table maintenance.Aug 30 2024, 2:22 PM

As part of T370898: [L] Track commons deletion requests the structured data team will be needing support for automatic table maintenance for their new Iceberg table.

Considering the table will be ~1M rows, and thus relatively small, the availability of this mechanism should not block going into production. However, we should make this mechanism available to them ASAP so that the performance of their lambda architecture doesn't degrade over time.

CC @Ahoelzl

Ahoelzl moved this task from Incoming (new tickets) to Backlog on the Data-Engineering board.Oct 23 2024, 8:33 PM

xcollazo added a subtask: T391135: Iceberg table maintenance for tables under wmf_product database.Apr 4 2025, 5:35 PM

Perhaps we should do this work jointly with T383931: Unify Airflow's datasets.yaml config files across instances?

So, now that T383931 has been completed, what else is there that should be done with regards to this ticket? We now do have all the Iceberg table maintenance configuration available in a single file shared by all instances, plus the actual maintenance DAG that reads this configuration and executes on it is running in our main instance and takes care of all configured tables.

In T373693#10906965, @amastilovic wrote:

So, now that T383931 has been completed, what else is there that should be done with regards to this ticket? We now do have all the Iceberg table maintenance configuration available in a single file shared by all instances, plus the actual maintenance DAG that reads this configuration and executes on it is running in our main instance and takes care of all configured tables.

Roughly, we have to:

Move the table_maintenance DAG to wmf_airflow_common
Modify DAG code to be aware of the instance it is being run on.
Make it so that each instance runs table maintenance for the tables that its service user owns. So, say, the research Airflow instance (which is run with the analytics_research service user) would run maintenance for all of their tables, and no other. So we will filter the datasets.yaml by the produced_by tag.