As we scale up the amount of dumped wikis, we might find that the database itself needs scaling up, as it takes in more and more TPS.
Description
Description
Details
Details
Related Changes in Gerrit:
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | None | T88728 Improve Wikimedia dumping infrastructure | |||
| Resolved | BTullis | T352650 WE 5.4 KR - Hypothesis 5.4.4 - Q3 FY24/25 - Migrate current-generation dumps to run on kubernetes | |||
| Resolved | brouberol | T388378 Orchestrate dumps v1 from an airflow instance | |||
| Resolved | brouberol | T390945 Run an experimental dump of 200 regular sized wikis | |||
| Resolved | BTullis | T391564 Adjust the database resource requirements |
Event Timeline
This comment was removed by BTullis.
Comment Actions
Change #1161879 had a related patch set uploaded (by Btullis; author: Btullis):
[operations/deployment-charts@master] airflow-test-k8s: bump the max_connections
Comment Actions
The DB connection count is climbing to around 67% during a full-size dumps run on the test-k8s instance.
This is below our current limit of 100, but we can increase the limit to give ourselves more headroom.
Comment Actions
Change #1161879 merged by jenkins-bot:
[operations/deployment-charts@master] airflow-test-k8s: bump the max_connections
Comment Actions
Change #1163329 had a related patch set uploaded (by Brouberol; author: Brouberol):
[operations/deployment-charts@master] postgresql-airflow-test-k8s: rename value file
Comment Actions
Change #1163329 merged by Brouberol:
[operations/deployment-charts@master] postgresql-airflow-test-k8s: rename value file