We are going to execute many dump tasks, and they don't return anything we need to save to the database. Let's disable XCOMs for these tasks, which might save some DB storage space over time.
Description
Description
Details
Details
Related Changes in GitLab:
| Title | Reference | Author | Source Branch | Dest Branch | |
|---|---|---|---|---|---|
| test_k8s/dumpsv1: ensure dump tasks don't store their result to the database | repos/data-engineering/airflow-dags!1223 | brouberol | T391481 | main |
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | None | T88728 Improve Wikimedia dumping infrastructure | |||
| Resolved | BTullis | T352650 WE 5.4 KR - Hypothesis 5.4.4 - Q3 FY24/25 - Migrate current-generation dumps to run on kubernetes | |||
| Resolved | brouberol | T388378 Orchestrate dumps v1 from an airflow instance | |||
| Resolved | brouberol | T390945 Run an experimental dump of 200 regular sized wikis | |||
| Resolved | brouberol | T391481 Disable XCOMs for dump tasks |
Event Timeline
Comment Actions
brouberol merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1223
test_k8s/dumpsv1: ensure dump tasks don't store their result to the database