We'll need a custom configuration for cloudelastic hosts (specifically, we need more SATA SSDs for storage). See T233720 for our last order. Per #wikimedia-dcops IRC conversation with @RobH , our next step is to open a procurement ticket as described on DC Ops' Phab group page .

Cloudelastic has too many disks. Elastic does not need fault tolerance at the filesystem/RAID level, so we should be able to cut back to RAID 0 and save some money.

Currently, cloudelastic uses 6x 1.9TB in RAID 10, which gives us ~5 TB usable space (minus quite a bit for reserved blocks, OS, etc). Filesystem usage (starting in Oct 2021 in Grafana) starts at 1.8T (~46%) and is now up to 1.9T (~54%). Naively, we'll place the growth rate at .1 TB/ per 18 months or .33 TB during the planned five-year lifespan of the hosts.

Considerations around Elastic and disk space usage:

• At 85% disk usage, elastic will stop accepting new indices on the node
• at 90%, it will try to relocate existing shards away from the node
• at 95%, it goes read only.
• Storage usage increases during failover. During a failover event ( T330165 ) in which we lost 2 nodes, usage grew to 3.15 TB (77%). (you can see this in Grafana ).

So we need to have enough disk space to stay below the 85% threshold, accounting for both growth and failover events.

Possibilities (RAID-0):

• 3x 1.9 TB SSD (Similar to what we do on production Elastic hosts). Assuming 0% reserved blocks and ~200 GB for OS, this yields ~5.2TB storage for elastic data on the cloudelastic host. Assuming the .33 TB growth rate, a failover event would raise disk usage to around ~3.48 , or 63% of usage.
• 4x 1.9 TB SSD. After removing the default 5% reserved blocks, this yields ~7 TB storage for elastic data on the cloudelastic host. Assuming the .33 TB growth rate, a failover event would raise disk usage to around ~3.48 , or 50% of usage.

Cost savings? I'm not sure exactly, but retail price for these drives is listed at ~$1400 USD Assuming we went with 3x drives and paid retail price, we'd save around $16,800 from the next order (4 servers) , or ~$11,200 if we went with 4x drives.

Team, please double-check my calculations and let me know if you have any objections or feedback.

Things we looked at

Index store size (gives an idea of growth in storage requirements)

https://grafana-rw.wikimedia.org/d/fomV6x0Gz/elasticsearch-index-stats?orgId=1&var-index=commonswiki_file&var-exported_cluster=production-search-eqiad&from=now-2y&to=now&editPanel=12

^ modified to sum(max by (index) (elasticsearch_indices_store_size_bytes_primary{exported_cluster="$exported_cluster", index=~".*"}))

Conclusion: No major spikes in index size over the last year.

Full-text QPS

https://grafana-rw.wikimedia.org/d/000000455/elasticsearch-percentiles?forceLogin&from=now-5y&orgId=1&refresh=1m&to=now&var-cirrus_group=eqiad&var-cluster=elasticsearch&var-exported_cluster=production-search&var-smoothing=1
^ modified to summarize(sum(MediaWiki.CirrusSearch.{eqiad,codfw}.requestTimeMs.full_text.sample_rate), "1y","avg")

Conclusion: No major growth in QPS over the last year

p75 request response time

https://grafana-rw.wikimedia.org/d/000000455/elasticsearch-percentiles?forceLogin&from=now-5y&orgId=1&refresh=1m&to=now&var-cirrus_group=eqiad&var-cluster=elasticsearch&var-exported_cluster=production-search&var-smoothing=1&viewPanel=20

Our 75th percentile request time performance is still acceptable. We see a doubling in time required over the last several weeks but this is an artifact of recent datacenter switchover where eqiad is doing all of the work instead of it being shared between eqiad and codfw.

In conclusion, we'll want to request a small increase in Elasticsearch server count in order to stay ahead of future growth, but we do not need to do a large scale-up like we did in the last year (scaling up our server capacity by 50%). I'm looking at a number of about 10% growth in server counts which corresponds to 5 new eqiad hosts and 5 new codfw hosts (10 hosts total).

Per conversation with @dcausse , we need to forecast resource usage for the search platform streaming updater and determine whether or not we need to contribute hosts to the Wikikube cluster.