The dag drop_old_data_daily has been disabled while migrating the airflow scheduler to k8s, we should adapt it (probably by updating the --execute hash) and re-enable it.
AC:
| Title | Reference | Author | Source Branch | Dest Branch | |
|---|---|---|---|---|---|
| search: update drop_old_data checksums | repos/data-engineering/airflow-dags!1160 | dcausse | T386097-search-update-drop-old-data-checksums | main |
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | brouberol | T362788 Migrate Airflow to the dse-k8s cluster | |||
| Resolved | brouberol | T364389 Migrate the airflow scheduler components to Kubernetes | |||
| Resolved | bking | T380621 Migrate the airflow-search scheduler to Kubernetes | |||
| Resolved | dcausse | T386097 Re-enable drop_old_data_daily in airflow-search |
@dcausse I had a look at the tables managed by this DAG, and none is declared as EXTERNAL. The dag (correctly) drops both partitions from Hive metastore, and data from hdfs.
dcausse opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1160
search: update drop_old_data checksums
dcausse merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1160
search: update drop_old_data checksums