When deploying new DAGs with dynamic components on our Kubernetes-based Airflow instances, we should optimize the resource requests and limits for better cluster efficiency.Refine currently generates a high overhead for relatively small work units:
* Many templated calls generating individual requests/jobs.
* Small streams processed as separate tasks instead of being grouped.
* Precomputation of all configuration in configuration step and storage in xcoms
Currently, we are processing 197 streams, each generating 6 tasks (including a sensor and 2 Spark tasks), totaling 1,182 task instances per hour. To manage this load efficiently, we can follow the resource configuration pattern recently applied to the Canary events DAG:This leads to:
* Large numbers of task instances (200*3=600 tasks per run).
* pressure on DB (large rows + number of rows)
* Increased dependency check times.
https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/blob/571d7f54114fb61bc101df525b369ea6e4aff71e/main/dags/canary_events/canary_events_dag.py#L120First improvements:
* Run small jobs within pod (local spark) as it's done in analytics-test
* Only store in xcoms the minimal information needed to launch downstream tasks T375064
* Compute configuration at execution time instead of precomputing all in configuration step
We’ll start by testing medium and low resource profiles and adjust based on observations from:
• [[ https://grafana.wikimedia.org/d/pz5A-vASz/kubernetes-resources | Kubernetes Resources dashboard ]]
• [[ https://grafana.wikimedia.org/d/lFXqBGxWk/airflow-instance | Airflow Instance dashboard ]]
• Pod-level stats
Eventually, later, we can consider batching. We would loose some visibility on individual items but
we would reduce DB pressure and number of task instances.
Huge thanks to @brouberol for the guidance!