The datahub airflow plugin documentation shows an example of how to setup a DAG that deletes the obsolete data out of Datahub.
As it turns out, we do have a DAG with a typo in Datahub, so we could take this as an opportunity to test the cleanup DAG.
Description
Description
Details
Details
Related Changes in GitLab:
| Title | Reference | Author | Source Branch | Dest Branch | |
|---|---|---|---|---|---|
| Clean up obsolete DAGs from Datahub | repos/data-engineering/airflow-dags!872 | stevemunene | T375714_dag_cleanup | main |
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | brouberol | T362788 Migrate Airflow to the dse-k8s cluster | |||
| Resolved | brouberol | T375715 Add maintenance DAGs to wmf_airflow_common | |||
| Resolved | • Stevemunene | T375714 Cleanup obsolete DAGs from Datahub |
Event Timeline
Comment Actions
stevemunene opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/872
Draft: Clean up obsolete DAGs from Datahub
Comment Actions
stevemunene merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/872
Clean up obsolete DAGs from Datahub