Running 2 dumps per month will create about 2x60k tasks in database with associated XCOMs, logs, etc.
This means that we should expect about 1.5M tasks a year in our metadstore. Should we keep everything? We need to determine our database retention to make sure we're not clogging down to database, while keeping appropriate backfill/investigation capabilities.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | None | T88728 Improve Wikimedia dumping infrastructure | |||
| Resolved | BTullis | T352650 WE 5.4 KR - Hypothesis 5.4.4 - Q3 FY24/25 - Migrate current-generation dumps to run on kubernetes | |||
| Resolved | brouberol | T388378 Orchestrate dumps v1 from an airflow instance | |||
| Resolved | brouberol | T390945 Run an experimental dump of 200 regular sized wikis | |||
| Resolved | BTullis | T391791 Adjust database retention settings |
I think we can close this, really.
After T392461: Batch dump jobs within a single airflow task to speed up DAG execution was completed, we found that the number of task instances per dump is significantly smaller than first estimated.
So I'm OK with leaving the database retention to the default, for now. If we start seeing database bloat, then we can come back to it.