I've noticed that the dump DAGs dependencies are current setup in a way that each task in the dump task group has a direct upstream dependency on the fetch_job_pod_spec task.
We should make sure that the DAG task dependencies are simplified as such:
fetch_job_pod_spec --[--> dump_batch_1 -> sync_batch_1 -> ... -> dump_batch_n -> sync_batch_n ]
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | None | T88728 Improve Wikimedia dumping infrastructure | |||
| Resolved | BTullis | T352650 WE 5.4 KR - Hypothesis 5.4.4 - Q3 FY24/25 - Migrate current-generation dumps to run on kubernetes | |||
| Resolved | brouberol | T388378 Orchestrate dumps v1 from an airflow instance | |||
| Resolved | brouberol | T393407 Untangle dump DAG dependencies |
There's no guarantee, but this might have a side-effect of lowering the scheduler load as well, as the runtime dependencies checks would be simplified.