We need to decide if and how extra capacity we want to budget for in FY 26/27. To do that, we need to estimate our capacity needs in 15 months' time (i.e. the end of 26/27 Q4, or June 2027). This is challenging, because over the last 12 months (about as far back as grafana will easily let us plot), growth has been distinctly non-linear -
[the green line is eqiad, the yellow codfw - click on the image to get a more usefully-sized graph, and note the non-zero Y axis]
To put that in concrete terms, make estimates of annual growth rate based on 12,6, and 2 months ago, with current usage at 1.23pB in codfw, 1.26pB in eqiad:
| Months ago | codfw size (tB) | codfw growth since (tB) | codfw growth rate (tB/y) | eqiad size (tB) | eqiad growth since (tB) | eqiad growth rate (tB/y) |
|---|---|---|---|---|---|---|
| 12 | 938 | 292 | 292 | 1017 | 243 | 243 |
| 6 | 992 | 238 | 476 | 1070 | 190 | 380 |
| 2 | 1120 | 110 | 660 | 1180 | 80 | 480 |
We use 3x replication, so these figures need multiplying by 3 to get raw capacity. Currently codfw has 33 storage hosts (so 6336TB capacity), and eqiad has 34 (so 6528TB), and capacity is about 73% used. We'd like to not plan to exceed 85% capacity.
The numbers of objects are declining because of on-going work to delete thumbnails (T379942), but that capacity gain is more than offset by uploads of more and larger originals.
If we take the most pessimistic projection of 660TB/year, then over 15 months that equates to 660*15*3/12 = 2475TB in raw storage.
Current total raw storage use in eqiad (per swift-recon -d) is 4230 TB. Adding 2475 gives 6705 TB. If we want that to be 85% of total capacity, we need total capacity to be 7888 TB, which is just over 41 Config-J systems, 7 more than we currently have.
Current total raw storage use in codfw is 4254 TB. Adding 2475 gives 6729 TB. If we want that to be 85% of total capacity we need total capacity to be 7916TB, which is likewise just over 41 Config-J systems, 8 more than we currently have.