The current implementation of the Airflow DAGs for Refine generates significant overhead, with approximately 1500 task instances per run. This overhead strains the limited resources of our current Airflow deployment, impacting the system’s overall efficiency and scalability.

A substantial portion of this overhead stems from the preparation steps, which create custom arguments for each mapped task. While this approach offers benefits such as explicitness, alignment with Airflow documentation, and ease of debugging, it introduces unnecessary complexity and resource consumption.

Proposed Solution:

1. Precompute Parameters: Move the preparation of all task parameters to the first task of the DAG. This task will handle configuration and compute the necessary parameters.
2. Use XComs: Leverage XComs to pass precomputed parameters to downstream tasks, eliminating the need for individual preparation steps for each mapped task.
3. Test the updated DAG for correctness and performance improvements.

Benefits:

1. Reduced number of task instances per DAG run, decreasing resource usage.
2. Improved performance and scalability of Airflow deployment.
3. Simplified DAG structure

Cons: Loss of explicitness in individual task configurations. Mitigation: Tests & doc.