I worked through this with claude and managed to build a local reproduction of the issue along with a potential resolution. Our initial theory was cluster state size, but it looks that that is not that case. Full findings stored P95921, the summary is:

The master election storms are **not** primarily caused by cluster state size. They
are caused by a per-node-join CPU cost that scales with the number of indices
carrying `auto_expand_replicas`, via a code path that **does not exist in
OpenSearch 1.3.x**. State size makes it worse, but is not the trigger.

I'm now working on a fix in CirrusSearch, we don't actually take advantage of auto_expand_replicas, we were simply using it to have a single code path for both us and small clusters outside WMF. We actually set these to constant values anyways (auto_expand_replicas: 2-2).

Mentioned in SAL (#wikimedia-operations) [2026-08-05T20:43:16Z] <ebernhardson> T434008: changing cloudelastic:9643 from auto_expand_replicas to number_of_replicas

We are testing the above fix on cloudelastic and will apply to production ASAP if it seems to work.

The fix involves changing index settings to hard-code the number of replicas instead of asking the cluster to figure out the number of replicas for us. This can be safely applied to existing indices , so we're basically doing

curl -XPUT https://cluster:9243/foo_content,foo_general,blah_content,blah_general/_settings -d '{"index":{"auto_expand_replicas": "false", "number_of_replicas": 2}}'

in all indices, in all clusters. We are taking a phased approach and will confirm on cloudelastic, then the smaller clusters (omega,psi) before moving on to the main production clusters (chi). We will also depool the DC before applying on any prod clusters so we minimize user impact.

I'm happy to report that all tests were successful, and we have applied the changes to all production clusters. After successful tests in cloudelastic, I depooled the EQIAD psi and chi clusters and restarted the active master. Both clusters recovered instantly, a marked difference from earlier today when we had multiple outages after restarting the master-eligibles.

I'll keep the ticket open until we can make a note in our docs about this.

Change #1321635 had a related patch set uploaded (by Ebernhardson; author: Ebernhardson):

[mediawiki/extensions/CirrusSearch@master] Use number_of_replicas where possible

https://gerrit.wikimedia.org/r/1321635

In T434008#12189137, @EBernhardson wrote:

I worked through this with claude and managed to build a local reproduction of the issue along with a potential resolution. Our initial theory was cluster state size, but it looks that that is not that case. Full findings stored P95921, the summary is:

The master election storms are **not** primarily caused by cluster state size. They
are caused by a per-node-join CPU cost that scales with the number of indices
carrying `auto_expand_replicas`, via a code path that **does not exist in
OpenSearch 1.3.x**. State size makes it worse, but is not the trigger.

I'm now working on a fix in CirrusSearch, we don't actually take advantage of auto_expand_replicas, we were simply using it to have a single code path for both us and small clusters outside WMF. We actually set these to constant values anyways (auto_expand_replicas: 2-2).

I checked opensearch 3.8.0, the latest release. I does still suffer from this problem, but at a 50-100x reduced cost. Where my reproduction would storm for 280s on 2.19.5, it only storms for ~3s on 3.8.0. Should be irrelevant though, our fix also resolves the problem when running against 3.8.0.

Change #1321635 merged by jenkins-bot:

[mediawiki/extensions/CirrusSearch@master] Use number_of_replicas where possible

https://gerrit.wikimedia.org/r/1321635