The working theory is that I set node_concurrent_recoveries too high (40, normal setting is 4) and cloudelastic1010 volunteered to take too many shards and fell over.

Note that we also had shards getting stuck even when I set node_concurrent_recoveries, although the cluster never dipped into the red.

Next steps: check disk config/health of cloudelastic1010. If that checks out, we might also try banning a node again and seeing how each host responds in terms of I/O pressure.

Mentioned in SAL (#wikimedia-operations) [2026-04-29T14:37:09Z] <inflatador> bking@cloudelastic1010 run smartctl against all physical disks T424852

Change #1282377 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] cirrussearch: install atop utility

https://gerrit.wikimedia.org/r/1282377

Change #1282383 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] standard_packages: prevent atop package from automatic purges

https://gerrit.wikimedia.org/r/1282383

All in favor of installing any tool necessary for profiling/debugging a specific issue, absolutely no problems there, but I would talk to Observability team about the gaps in observability in the long term, as existing tools are supposed to cover all essential needs regarding basic monitoring, and standarization would help everyone and prevent multiple tools doing the same thing.

Change #1282399 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] cloudelastic: explicitly disable security plugin

https://gerrit.wikimedia.org/r/1282399

Change #1282383 merged by Bking:

[operations/puppet@production] standard_packages: prevent atop package from automatic purges

https://gerrit.wikimedia.org/r/1282383

Change #1282399 merged by Bking:

[operations/puppet@production] cloudelastic: explicitly disable security plugin

https://gerrit.wikimedia.org/r/1282399

Change #1282409 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] cloudelastic: remove systemd override that uses PrivateMounts

https://gerrit.wikimedia.org/r/1282409

Change #1282409 merged by Bking:

[operations/puppet@production] cloudelastic: Disable systemd override that uses PrivateMounts

https://gerrit.wikimedia.org/r/1282409

Mentioned in SAL (#wikimedia-operations) [2026-05-04T19:23:08Z] <bking@cumin2002> START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: remove privatemounts to see if it helps - bking@cumin2002 - T424852

Mentioned in SAL (#wikimedia-operations) [2026-05-04T19:23:36Z] <bking@cumin2002> END (ERROR) - Cookbook sre.elasticsearch.rolling-operation (exit_code=97) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: remove privatemounts to see if it helps - bking@cumin2002 - T424852

Icinga downtime and Alertmanager silence (ID=76eb216d-a898-4f58-8a56-4198503d8105) set by bking@cumin2002 for 1 day, 0:00:00 on 6 host(s) and their services with reason: ongoing troubleshooting

cloudelastic[1007-1012].eqiad.wmnet

Mentioned in SAL (#wikimedia-operations) [2026-05-04T19:37:07Z] <bking@cumin2002> START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: remove privatemounts to see if it helps - bking@cumin2002 - T424852

Mentioned in SAL (#wikimedia-operations) [2026-05-04T19:40:25Z] <bking@cumin2002> END (ERROR) - Cookbook sre.elasticsearch.rolling-operation (exit_code=97) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: remove privatemounts to see if it helps - bking@cumin2002 - T424852

I tried rolling back the systemd changes, but the cluster is so unresponsive right now I'm not going to make any further changes. Here is what's left to completely undo privateMounts:

1. Restart all opensearch_2@ services on cloudelastic1007
2. Restart opensearch_2@chi service on cloudelastic1009.

We should leave this change in place for at least a few days to see if it makes a difference. Based on the I/O PSI metrics, I'm not seeing any changes.

We should also see the Streaming Updater backlog start to drop if the performance improves.

@EBernhardson helped troubleshoot this today. It seems that we may be affected by the readahead issues he originally discovered in T264053.

The cluster was red almost all day due to I/O contention basically grinding cloudelastic1008 and cloudelastic1010 to a halt. The presumed fix for this was to manually set readahead commands like sudo blockdev --setra 16 /dev/md2, then restart opensearch to force the memory to be remapped with lowered readaheads. This seems to have greatly increased I/O performance .

This is a bit concerning since we already have a service running (opensearch-madvise) that should be doing this for us automatically. We'll investigate more tomorrow. As of now, I have rolled back several other one-off changes (such as disabling nginx and privatemounts).

Tomorrow when I get in, I'll try another rolling restart of the cluster to see if the cluster can handle losing a node for a few moments without going red.

Change #1283083 had a related patch set uploaded (by Bking; author: Bking):

[operations/alerts@master] data-platform: Add alerts for cirrus memory or I/O stalls

https://gerrit.wikimedia.org/r/1283083

Change #1283083 merged by jenkins-bot:

[operations/alerts@master] data-platform: Add alerts for cirrus memory/IO stalls

https://gerrit.wikimedia.org/r/1283083