We need to periodically remove data stored in HDFS and PVC by data_reload_dag pipeline.

A pattern adopted is airflow-dag is to use a daily companion dag that drops data older than a certain threshold (based on the dataset partitions).

An example for wdqs v1 is available at https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/blob/main/search/dags/drop_old_data_daily.py?ref_type=heads#L92.

However, this dag relies on data deletion helper function from the refinery repo that we found out are not compatible with the partition scheme used in WDQS hdfs datasets.
For details: T431438: discovery.wikibase_rdf_subgraphs (/wmf/data/discovery/wikidata/rdf_subgraphs) is using 34Tb and is not properly cleaned up

We need to review existing approaches and validate if they would work for our use case, and if not find alternatives.

Things to explore:

• Adapting the existing cleanup to work with our non-Iceberg tables
• Writing our own data cleanup (probably not advisable)
• Upgrading our tables to Iceberg (probably hard because of the necessary spark version bumps)
• ...

AC

• Recommendation for data retention plan
• phab filed to track it