The worker dump script iterates over all kinds of dumps if not provided the --job <name-csv> value.
Instead of letting it handle the whole bulk of the work, we will generate an airflow task for each job, so that we get a finer grain view of where we are.
This will also help us uploading the result of each dump to the clouddumps servers as soon as it is generated.
| Title | Reference | Author | Source Branch | Dest Branch | |
|---|---|---|---|---|---|
| test_k8s/dumpsv1: Include the dump job name in the pod name | repos/data-engineering/airflow-dags!1211 | brouberol | T390852 | main | |
| Split dump task into a chain of tasks, one per dump job type | repos/data-engineering/airflow-dags!1210 | brouberol | T390852 | main |
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | None | T88728 Improve Wikimedia dumping infrastructure | |||
| Resolved | BTullis | T352650 WE 5.4 KR - Hypothesis 5.4.4 - Q3 FY24/25 - Migrate current-generation dumps to run on kubernetes | |||
| Resolved | brouberol | T388378 Orchestrate dumps v1 from an airflow instance | |||
| Resolved | brouberol | T390852 Split wiki dump DAG into multiple tasks |
brouberol merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1210
Split dump task into a chain of tasks, one per dump job type
brouberol merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1211
test_k8s/dumpsv1: Include the dump job name in the pod name