We've seen warnings for >50GB shards for commonswiki_file, and while we're at it we want to do a quick scan to see if other shard types are nearing the limit.
Shard count configuration lives here: https://github.com/wikimedia/operations-mediawiki-config/blob/master/wmf-config/InitialiseSettings.php#L23391
The intention is to buy us hopefully 2 years or so, at which point we should have 10G networking on our Elasticsearch cluster which would let us start using higher shard counts because rebalancing shards across nodes (for example after a node drops offline) is ratelimited by network throughput.
The tradeoff here is, as a general rule, bigger shards mean less overhead, but are slower to rebalance due to the above.
In addition to the shard tuning, we should also tune the "rescore":
we run a second phase of scoring over the top N documents per shard, and now there will be ~40% more shards, so 40% more docs in the rescore phase
CODE CHANGES
- Tune the shard counts
- Tune the rescore
- Make replica counts same between eqiad/codfw everywhere given that they now have equal # of servers (36 each).
- Functionally this means enwiki_content and dewiki_content for codfw need replica count bumped 2->3
- Bonus: Created https://wikitech.wikimedia.org/wiki/Search#Tuning_Shard_Counts with the general reasoning/tradeoffs on shard count decisionmaking
DEPLOYMENT