| brouberol | |
| Apr 12 2025, 1:12 PM |
| F59060807: Screenshot 2025-04-14 at 13.48.27.png | |
| Apr 14 2025, 11:49 AM |
| F59060685: Screenshot 2025-04-14 at 12.53.11.png | |
| Apr 14 2025, 10:54 AM |
Now that we've statically defined the dumps DAG, we've seen DAG processing times jump from ~1s to ~7s with 32 wikis. We need to adjust the file processing timeout.
I suggest setting it to a super high value, measure how much time it really takes the scheduler to parse the DAG with all the wikis, and adjust the value more realistically.
| Subject | Author | Repo | Branch | Lines +/- | |
|---|---|---|---|---|---|
| airflow-test-k8s: adjust dag/file processing timeout to account for large v1 dumps dags | Brouberol | operations/deployment-charts | master | +2 -0 |
| Title | Reference | Author | Source Branch | Dest Branch | |
|---|---|---|---|---|---|
| test_k8s/dumpsv1: define the list of wikis in JSON files | repos/data-engineering/airflow-dags!1234 | brouberol | T391744 | main |
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | None | T88728 Improve Wikimedia dumping infrastructure | |||
| Resolved | BTullis | T352650 WE 5.4 KR - Hypothesis 5.4.4 - Q3 FY24/25 - Migrate current-generation dumps to run on kubernetes | |||
| Resolved | brouberol | T388378 Orchestrate dumps v1 from an airflow instance | |||
| Resolved | brouberol | T390945 Run an experimental dump of 200 regular sized wikis | |||
| Resolved | brouberol | T391744 Increase file processing timeout |
An initial test of importing the DAG with all 1008 wikis shows an import time of 3 min 33s.
I think we can safely set
dag_file_processor_timeout: 300 dagbag_import_timeout: 300
Change #1136351 had a related patch set uploaded (by Brouberol; author: Brouberol):
[operations/deployment-charts@master] airflow-test-k8s: adjust dag/file processing timeout to account for large v1 dumps dags
I've also experimented with using chain(tg_tasks) instead of using my manual wiring, and it seems to be slower, by ~30s. I'm going to stick to the faster implementation.
Change #1136351 merged by Brouberol:
[operations/deployment-charts@master] airflow-test-k8s: adjust dag/file processing timeout to account for large v1 dumps dags
brouberol merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1234
test_k8s/dumpsv1: define the list of wikis in JSON files