We will be creating a lot of Pods while the dumps are ongoing (although, this will be limited by the airflow pool parallelism setting).
We need to keep an eye on the load of the Kubernetes API masters, to make sure we're not causing cluster-wide performance degradations.
Description
Description
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | None | T88728 Improve Wikimedia dumping infrastructure | |||
| Resolved | BTullis | T352650 WE 5.4 KR - Hypothesis 5.4.4 - Q3 FY24/25 - Migrate current-generation dumps to run on kubernetes | |||
| Resolved | brouberol | T388378 Orchestrate dumps v1 from an airflow instance | |||
| Resolved | BTullis | T389778 Monitor the load on the Kubernetes API masters |
Event Timeline
Comment Actions
We have monitored the first runs of full-sized dumps runs and it seems that the load on the Kubernetes masters is within tolerance.
There are some issues around scheduling timeouts for pods and some latencies in T396738: Increase threshold for KubeletOperationalLatency for dse-k8s-eqiad but I think that we can address those independently of this ticket.