Due to critical heap usage I'll be limiting cluster.max_shards_per_node to 800. This is still well above Elastic's recommendation of 640 shards for 32GB of heap. When we run out of shards in the future we can incrementally increase it as long as the heap usage remains within reason. We need to add additional data nodes when we can no longer increase the limit or heap size.

In T350404#9315836, @Andrew-WMDE wrote:

The root cause behind the index failures is that we have reached our cluster's shard limit:

⧼Validation Failed: 1: this action would add [2] total shards, but this cluster currently has [2000]/[2000] maximum shards open;⧽

Our temporary work around was to increase cluster.max_shards_per_node to 1200 and rerun the indexing jobs for any instances that failed.

https://www.elastic.co/guide/en/elasticsearch/reference/7.17/size-your-shards.html#_this_action_would_add_x_total_shards_but_this_cluster_currently_has_yz_maximum_shards_open

Waiting on T350404

• Manually run index creation jobs for instances where it failed

Waiting on T335894

A better long-term solution might be to add an additional data node on production.

The root cause behind the index failures is that we have reached our cluster's shard limit:

Waiting to evaluate new cluster with real-world usage

Waiting for indexing jobs to finish

Waiting on T335891