| amastilovic | |
| Jun 12 2025, 1:14 AM |
| F62306502: image.png | |
| Jun 12 2025, 9:25 PM |
| F62305297: image.png | |
| Jun 12 2025, 6:33 PM |
| F62304454: image.png | |
| Jun 12 2025, 4:34 PM |
| F62304450: image.png | |
| Jun 12 2025, 4:34 PM |
Diagnosis: Airflow scheduler on main Kubernetes deployment is configured to be able to run 128 tasks in parallel (across all DAGs) but the actual number of tasks running or queued for execution usually stays at 40 and below. Even when the scheduler schedules over a thousand tasks for execution, all 128 are never utilized all at once.
Metrics clearly show that the scheduler's pool of running tasks is under-utilized even when there's over a thousand scheduled tasks:
I have just applied this change: Bump up the CPU and RAM resources for airflow related namespaces
So let's see if it makes a difference to the parallelism of your tasks.
https://grafana.wikimedia.org/goto/CYeMNYLNR?orgId=1
Change #1161047 had a related patch set uploaded (by Aqu; author: Aqu):
[operations/deployment-charts@master] Airflow analytics-test: Optimization for LocalExecutors
I've added a pod counts graph here: https://grafana.wikimedia.org/d/ca9c0221-4a0d-4833-865b-f14a3e813c97/kubernetes-resource-quotas?orgId=1&from=now-6h&to=now&timezone=utc&var-site=eqiad&var-prometheus=k8s-dse&var-namespace=airflow-main
It's more accurate as short tasks don't have the time to increment the level of running tasks within the Airflow db.
With the graph we can see peaks of parallelization matching the level in our config.
Change #1161047 merged by Brouberol:
[operations/deployment-charts@master] Airflow analytics-test: Optimization for LocalExecutors
Change #1165900 had a related patch set uploaded (by Btullis; author: Btullis):
[operations/deployment-charts@master] Increase max_tis_per_query for airflow-main
Change #1165900 merged by jenkins-bot:
[operations/deployment-charts@master] Increase max_tis_per_query for airflow-main
Since refine_to_hive has been moved into its own queue it lets some chances to other dags tasks to be processed.
