The following experiment shows 2 subsequent DAG runs for aawikionly. The first one is without absolute weight rule, and the second one is with.

I'm not 100% sure whether enabling absolute weight rules has made the DAG faster or not. I'm going to delete the whole data on ceph, restart the DAG without absolute weight rules, and start all over again with the weight rules.

What we see though is that the DAG dependency checks are clearly responsible for the scheduler loop time increasing. Which means that the DAG is "complex". That probably ties into T391483 more than this ticket though.

I think that the effect of the fixed priority should show when inserting a large DAG, aka at the very start. A DAG of a 30 tasks might not show quite the affect as would a DAG of 2000. That being said, we kind of see an effect of using absolute weights, which reduces the initial peak of scheduling loop time on the 1st and 3rd experiments (without absolute weights).

I'll now experiment with more wikis.

What is becoming clear is that using an absolute weight rule allows the scheduler to perform its initial work faster (basically creating all the tasks with the appropriate metadata in DB). We've been seeing a ~40% scheduler loop time reduction when starting the DAG for 16 wikis when relying on absolute weights. It's not a game changer for such a long DAG, but we'll take it anyway.

brouberol merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1228

test_k8s/dumpsv1: set an absolute weight on each dumps task to speed up scheduling