| brouberol | |
| Apr 23 2025, 8:44 AM |
| F59547356: Screenshot 2025-04-29 at 14.08.48.png | |
| Apr 29 2025, 12:09 PM |
As each task relies on the scheduling and execution of 2 pods (the task itself as well as the pod created by the KubernetesPodOperator), splitting each dump job into its dedicated airflow task incurs a steep price paid in waiting for pods to be scheduled.
It might be easier to batch jobs by, say, 5, to both keep the DAG small as well as speed up overall execution.
| Title | Reference | Author | Source Branch | Dest Branch | |
|---|---|---|---|---|---|
| test_k8s/dumpsv1: batch multiple dump jobs in the same airflow task | repos/data-engineering/airflow-dags!1273 | brouberol | T392461 | main |
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | None | T88728 Improve Wikimedia dumping infrastructure | |||
| Resolved | BTullis | T352650 WE 5.4 KR - Hypothesis 5.4.4 - Q3 FY24/25 - Migrate current-generation dumps to run on kubernetes | |||
| Resolved | brouberol | T388378 Orchestrate dumps v1 from an airflow instance | |||
| Resolved | brouberol | T390945 Run an experimental dump of 200 regular sized wikis | |||
| Resolved | brouberol | T392461 Batch dump jobs within a single airflow task to speed up DAG execution |
brouberol merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1273
test_k8s/dumpsv1: batch multiple dump jobs in the same airflow task