If we settle on using statically defined DAGs, the file processing time will go up. To keep scheduler resources low, we need to make sure that we don't process the DAG files too often, and also that we increase the file processing timeout.
Description
Description
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | None | T88728 Improve Wikimedia dumping infrastructure | |||
| Resolved | BTullis | T352650 WE 5.4 KR - Hypothesis 5.4.4 - Q3 FY24/25 - Migrate current-generation dumps to run on kubernetes | |||
| Resolved | brouberol | T388378 Orchestrate dumps v1 from an airflow instance | |||
| Resolved | brouberol | T390945 Run an experimental dump of 200 regular sized wikis | |||
| Resolved | brouberol | T391483 Experiment with disabling dynamic task mapping | |||
| Resolved | brouberol | T391678 Adjust the file processing interval |
Event Timeline
Comment Actions
Actually, we might not need this, as the DAGs are only re-processed if changed. Let's close this one.