Initial thoughts (from talking to team about this):

• Resource usage has multiple dimensions. The most obvious to ensure we have information on are on-disk size and memory usage. We will also want to consider query concurrency and where sharding might end up in the future, which will have an impact on how many cores are needed to satisfy requested load.

• Many stats will be covered by work in T265931 to load the index stats elasticsearch reports into prometheus for selected indices. These stats wont tell the whole picture though, concerns like how much memory is needed for the linux disk cache cannot be derived from this information.

• We can get more confidence in our plans by partially implementing them in the alternate dc (currently codfw). We can use tags in elasticsearch to limit the indices in question to a limited set of nodes. It seems unlikely we will be able to safely push the remaining indices off those nodes and into the remainder of the cluster, but it should be looked in to. We can then route traffic for those wikis to that DC and monitor performance. One difficulty is that new machines we are buying have 256G ram, but most of codfw is 128G

Looked into the dashboards we created to see the state is today, and how that data has changed since late november when we deployed stats collection:

• As expected, the wikipedia content indices (enwiki, dewiki, frwiki, viwki) are quite stable in terms of size and growth. Any growth over the last 90 days is minimal, some indices are up a little and others are down a little.

• wikidatawiki_content has suprisingly not really grown since data collection started (nov 26, 2020). The total size grew maybe 5%, but then retracted a bit.

• commonswiki_file grew dramatically over december and early january, investigated and mitigated in T271493. The mitigations are slow to apply though, Growth in the recorded timespan was from 7.5TB to 12.0TB, mitigation stopped the growth but we've only declined to 11.3TB in the meantime. Because of the large growth and isolated reason for the growth, we don't really have any estimate on what the typical growth of commonswiki would be. To verify the mitigation is working i checked dumps sizes, they are declining ~6%/wk which suggests it is reducing the size of indexed data.

• Estimates on # of cpu cores we need concurrently are quite low for all but enwiki. Many of our large but not largest (cebwiki_content, viwiki_content) see < 1 core necessary on average, and 10 at peak. Large but not excessivly popular indices, such as commonswiki_file, dewiki_content, frwiki_content and wikidata_wiki also need only 20-50 cores on average, peaking at around 100. enwiki_content averages 300, but with typical peaks up to 400 and on day at > 600 cores. Really these are not core counts, but estimates of cpu seconds used per second. The real need is probably 1.5 to 2x the reported core count.

Overall, a cluster that could run these indices today (not accounting for future growth) would be:

• ~260 cores. Peak usage on wikidata_content is 30, 100 on commonswiki_file. We can't actually run servers full out, with all cores 100%. Overestimate at 2x usage to account for utilization.
• ~20TB usable disk. Current usage is 14TB between the two. We require nodes to use no more than 75% of disk space, estimate at 70% disk utilization. It's plausible commonswiki is oversized here, it could plausibly decline by 3TB once mitigations are complete. Thats still 11TB for both, ~16TB accounting for utilization.
• ~2.1TB in-memory disk cache. We don't have strong proof about how much memory is needed for the disk cache, but we know that the current cluster operates well at 15% of indices in disk cache, and has increasing IO usage as we aproach 10%. Estimate at 15% of current on-disk size of 14TB.

A recent server build we ordered for the search clusters is:

• 256G memory (est 220G disk-cache)
• 2.4T usable disk
• 40 cores

A cluster of 10 of those machines would thus have:

• 2.2TB of disk cache
• 24TB usable disk
• 400 cores

While i think it's entirely possible that the index size a year from now will be less than it is today, as large files are replaced with truncated file_text and deleted docs are merged out of the segments (commonswiki grew from 8 to 12TB in two months, but we are pruning most of that back out), I would be wary of suggesting we shouldn't overprovision. Maybe something like 2-4 extra machines, hard to say.

I think that the estimate at 20TB is not enough to support the current shape of the indices, mainly because we have a giant index at 11Tb. Assuming worst case scenario (reindexing commonswiki_file will require another 11Tb) 33Tb is what is needed to reindex commonswiki_file without reaching the 75% watermark.
So adding +4 machines will increase the overall usable disk space to 33.6Tb so I would suggest at least 14 machines to comfortably support the current sizes. Now should we assume that the expected decrease of commonswiki_file due to file_text truncation and my over-pessimistic reindex scenario will compensate future growth? Hard to say.

I completely forgot about how it will behave while reindexing. Indeed we must account for two full copies of any one index as part of normal operations. While reviewing the previous order to see if it would make sense to add disks to these machines, I noticed the above spec for a recent server build is incorrect. Specifically the most recent machines are 2x1.75TB disks, with 3.4T usable on /srv. That brings 10 servers up to 34TB, matching the identified need.

Corrected recent server build spec:

• 256G memory (est 220G disk-cache)
• 3.4T usable disk
• 40 cores

Some other operational conditions we should expect:

• hardware fails. history says we must expect 1-2 machines can be out of the cluster for a month at a time.
• restarts typically run multiple instances at a time, we likely want to restart at least 2 instances at a time without impacting the service.

This seems sufficient justification to over provision to at least 12 instances, 14 still seems plausible. We might also want to consider what the chi cluster would no longer need, if perhaps 5 instances from the current clusters could be moved over (some are possibly up for refresh next year anyways?). That investigation would likely be a bit more complex than determining what the new cluster would require. Simpler, should budget allow, will be to allow that cluster to downsize as a normal part of hardware refreshes as we monitor its ongoing performance.

Sounds good to me, given the unpredictable growth of commons (current coverage of captions&depict statement is below 10%) I think it's sane to have at least 12 machines.