This will track progress on tools-infra quarterly "Analysis and automation on capacity usage and limitations"
We have most of the metrics we need for this already but they could use some organization and consolidation.
The deliverable for this is probably a short wiki page that links to grafana dashboards.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | None | T434128 Review capacity projection setup for toolforge and cloud-vps | |||
| Open | None | T434131 cloud-vps: review our CPU overprovisioning policy |
The top row of this dashboard provides a good overview of how full our metal infrastructure is.
If we want to be resilient against the failure of a single row (T424658) then we should revise our concept of 'full' to something a bit under 75% (or 150% for over-provisionable CPUs.)
Floating IPs don't grow organically; our use of them is largely capped, and in particular toolforge growth doesn't consume additional IPs.
This dashboard makes a more aggressive attempt to track and predict ceph usage. It gets upset whenever a new server comes online, though, so may need a bit of tuning.
initial thoughts...