[[ https://wikitech.wikimedia.org/wiki/Search/Elasticsearch_Administration#Clusters_(chi,_psi,_omega) | Search Platform currently runs 3 separate OpenSearch clusters, 2 per host. ]] . We split the cluster into three because we determined that maintaining cluster state becomes prohibitively expensive past around 5,000 shards. While this has helped performance, it adds complexity to the day-to-day management. Puppet code, cookbooks, and load balancer config are just a few places where we've been bitten.
While we shouldn't change a fast and stable system just because it's difficult to manage, I do think there's room to discuss changes.
Creating this ticket to:
[] Collect ideas on how to reduce complexity.
[] Discuss feasibility of each idea and decide whether or not to move forward.
Ideas (feel free to add yours):
- Re-integrate the three clusters into one
- Run all 3 clusters on each host
- Move each cluster to its own discrete VM or container