The migration of the Refine System to Airflow requires the support to create dynamic DAGs or Tasks within Airflow.
Refine Airflow execution will be driven by configuration going forward with variable execution tasks.
The implications of a dynamic dag vs. task mapping should be evaluated by their impact on Airflow metrics observability and other operational aspects like re-running a job.
References:
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | None | T307500 Airflow Hackathon (May 2022) | |||
| Open | None | T307505 Refine jobs should be scheduled by Airflow | |||
| Resolved | Antoine_Quhen | T356362 [Refine Refactoring] [Spike] Define a concept and provide a PoC for dynamic DAG execution in Airflow |
@Antoine_Quhen, Can we close this task? Link to the work done in to get canary events in airlfow?
Being bold and resolving. This has been done for canary events already, and will be done for Refine on Airflow too. T307505: Refine jobs should be scheduled by Airflow