Curator is failing on logstash2026.Logs since late April:
Output of `sudo systemctl status curator_actions_cluster_wide.service -l`:logstash2026
```
●$ sudo journalctl -e -n 100000 -u curator_actions_cluster_wide.service - OpenSearch Curator action cluster_wide | grep 'Failed to complete action'
Loaded: loaded (/lib/systemd/system/curator_actions_cluster_wide.service; static)Apr 29 00:42:53 logstash2026 curator[1203410]: 2024-04-29 00:42:53,671 ERROR Failed to complete action: delete_indices. <class 'KeyError'>: 'store'
Active: failed (Result: exit-code) since Sat 2024-05-04 00:43:35 UTC; 2h 2min agoApr 29 14:29:34 logstash2026 curator[1322941]: 2024-04-29 14:29:34,263 ERROR Failed to complete action: forcemerge. <class 'KeyError'>: 'store'
TriggeredBy: ● curator_actions_cluster_wide.timerMay 01 00:42:34 logstash2026 curator[1631611]: 2024-05-01 00:42:34,934 ERROR Failed to complete action: delete_indices. <class 'curator.exceptions.FailedExecution'>: Exception encountered. Rerun with loglevel DEBUG and/or check Elasticsearch logs for more information. Exception: NotFoundError(404, 'index_not_found_exception', 'no such index [logstash-default-1-7.0.0-1-2024.01.31]', logstash-default-1-7.0.0-1-2024.01.31, index_or_alias)
Docs: https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_stateMay 02 17:03:36 logstash2026 curator[1840558]: 2024-05-02 17:03:36,497 ERROR Failed to complete action: forcemerge. <class 'curator.exceptions.FailedExecution'>: Exception encountered. Rerun with loglevel DEBUG and/or check Elasticsearch logs for more information. Exception: ConnectionTimeout caused by - ReadTimeout(HTTPConnectionPool(host='127.0.0.1', port=9200): Read timed out. (read timeout=21600))
Process: 2244849 ExecStart=/usr/bin/curator --config /etc/curator/production-elk7-codfw.yaml /etc/curator/cluster_wide_actions.yaml (code=exited, status=1/FAILURE)
Main PID: 2244849 (code=exited, status=1/FAILURE)
CPU: 12.964s
May 04 00:43:35 logstash2026 curator[2244850]: Traceback (most recent call last):
May 04 00:43:35 logstash2026 curator[2244850]: File "/tmp/curator_python/3.9.2/lib/python3.9/site-packages/opensearchpy/connection/http_requests.py", line 191, in perform_request
May 04 00:43:35 logstash2026 curator[2244850]: File "/tmp/curator_python/3.9.2/lib/python3.9/site-packages/requests/sessions.py", line 703, in send
May 04 00:43:35 logstash2026 curator[2244850]: File "/tmp/curator_python/3.9.2/lib/python3.9/site-packages/requests/adapters.py", line 532,May 03 00:43:40 logstash2026 curator[2044019]: 2024-05-03 00:43:40,457 ERROR Failed to complete action: replicas. <class 'curator.exceptions.FailedExecution'>: Exception encountered. Rerun with loglevel DEBUG and/or check Elasticsearch logs for more information. in send
May 04 00:43:35 logstash2026 curator[2244850]: requests.exceptions.Exception: ConnectionTimeout caused by - ReadTimeout: (HTTPConnectionPool(host='127.0.0.1', port=9200): Read timed out. (read timeout=30))
May 04 00:43:35 logstash2026 curator[2244850]: 2024-05-04 00:43:35,224 ERROR Failed to complete action: replicas. <class 'curator.exceptions.FailedExecution'>: Exception encountered. Rerun with loglevel DEBUG and/or check Elasticsearch logs for more information. Exception: ConnectionTimeout caused by - ReadTimeout(HTTPConnectionPool(host='127.0.0.1', port=9200): Read timed out. (read timeout=30))
May 04 00:43:35 logstash2026 systemd[1]: curator_actions_cluster_wide.service: Main process exited, code=exited, status=1/FAILURE
May 04 00:43:35 logstash2026 systemd[1]: curator_actions_cluster_wide.service: Failed with result 'exit-code'.
May 04 00:43:35 logstash2026 systemd[1]: Failed to start OpenSearch Curator action cluster_wide.
May 04 00:43:35 logstash2026 systemd[1]: curator_actions_cluster_wide.service: Consumed 12.964s CPU time.9 08:59:56 logstash2026 curator[3378863]: 2024-05-09 08:59:56,177 ERROR Failed to complete action: forcemerge. <class 'curator.exceptions.FailedExecution'>: Exception encountered. Rerun with loglevel DEBUG and/or check Elasticsearch logs for more information. Exception: ConnectionTimeout caused by - ReadTimeout(HTTPConnectionPool(host='127.0.0.1', port=9200): Read timed out. (read timeout=21600))
```
Output of `sudo journalctl -u curator_actions_cluster_wide.service`:
```
May 04 00:43:35 logstash2026 curator[2244850]: 2024-05-04 00:43:35,220 WARNING PUT http://127.0.0.1:9200/ecs-alerts-2-1.11.0-6-2023,ecs-default-1-1.11.0-6-2024.08,ecs-default-1-1.11.0-6-2024.09,ecs-k8s-1-1.11.0-6-2023.23,ecs-k8s-1-1.11.0-6-2023.28,ecs->
May 04 00:43:35 logstash2026 curator[2244850]: Traceback (most recent call last):
May 04 00:43:35 logstash2026 curator[2244850]: File "/tmp/curator_python/3.9.2/lib/python3.9/site-packages/urllib3/connectionpool.py", line 467, in _make_request
May 04 00:43:35 logstash2026 curator[2244850]: File "<string>", line 3, in raise_from
May 04 00:43:35 logstash2026 curator[2244850]: File "/tmp/curator_python/3.9.2/lib/python3.9/site-packages/urllib3/connectionpool.py", line 462, in _make_request
May 04 00:43:35 logstash2026 curator[2244850]: File "/usr/lib/python3.9/http/client.py", line 1347, in getresponse
May 04 00:43:35 logstash2026 curator[2244850]: response.begin()
May 04 00:43:35 logstash2026 curator[2244850]: File "/usr/lib/python3.9/http/client.py", line 307, in begin
May 04 00:43:35 logstash2026 curator[2244850]: version, status, reason = self._read_status()
May 04 00:43:35 logstash2026 curator[2244850]: File "/usr/lib/python3.9/http/client.py", line 268, in _read_status
May 04 00:43:35 logstash2026 curator[2244850]: line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
May 04 00:43:35 logstash2026 curator[2244850]: File "/usr/lib/python3.9/socket.py", line 704, in readinto
May 04 00:43:35 logstash2026 curator[2244850]: return self._sock.recv_into(b)
May 04 00:43:35 logstash2026 curator[2244850]: socket.timeout: timed out
May 04 00:43:35 logstash2026 curator[2244850]: During handling of the above exception, another exception occurred:
May 04 00:43:35 logstash2026 curator[2244850]: Traceback (most recent call last):logstash1026
May 04 00:43:35 logstash2026 curator[2244850]: File "/tmp/curator_python/3.9.2/lib/python3.9/site-packages/requests/adapters.py", line 486, in send
May 04 00:43:35 logstash2026 curator[2244850]: File "/tmp/curator_python/3.9.2/lib/python3.9/site-packages/urllib3/connectionpool.py", line 799, in urlopen
May 04 00:43:35 logstash2026 curator[2244850]: File "/tmp/curator_python/3.9.2/lib/python3.9/site-packages/urllib3/util/retry.py", line 550, in increment
May 04 00:43:35 logstash2026 curator[2244850]: File "/tmp/curator_python/3.9.2/lib/python3.9/site-packages/urllib3/packages/six.py", line 770, in reraise
May 04 00:43:35 logstash2026 curator[2244850]: File "/tmp/curator_python/3.9.2/lib/python3.9/site-packages/urllib3/connectionpool.py", line 715, in urlopen
May 04 00:43:35 logstash2026 curator[2244850]: File "/tmp/curator_python/3.9.2/lib/python3.9/site-packages/urllib3/connectionpool.py", line 469, in _make_request
May 04 00:43:35 logstash2026 curator[2244850]: File "/tmp/curator_python/3.9.2/lib/python3.9/site-packages/urllib3/connectionpool.py", line 358, in _raise_timeout
May 04 00:43:35 logstash2026 curator[2244850]: urllib3.exceptions.ReadTimeoutError: HTTPConnectionPool(host='127.0.0.1', port=9200): Read timed out. (read timeout=30)
May 04 00:43:35 logstash2026 curator[2244850]: During handling of the above exception, another exception occurred:
May 04 00:43:35 logstash2026 curator[2244850]: Traceback (most recent call last):
May 04 00:43:35 logstash2026 curator[2244850]: File "/tmp/curator_python/3.9.2/lib/python3.9/site-packages/opensearchpy/connection/http_requests.py", line 191, in perform_request
May 04 00:43:35 logstash2026 curator[2244850]: File "/tmp/curator_python/3.9.2/lib/python3.9/site-packages/requests/sessions.py", line 703, in send
May 04 00:43:35 logstash2026 curator[2244850]: File "/tmp/curator_python/3.9.2/lib/python3.9/site-packages/requests/adapters.py", line 532, in send
May 04 00:43:35 logstash2026 curator[2244850]: requests.exceptions.ReadTimeout: HTTPConnectionPool(host='127.0.0.1', port=9200): Read timed out. (read timeout=30)
May 04 00:43:35 logstash2026 curator[2244850]: 2024-05-04 00:43:35,224 ERROR Failed to complete action: replicas. <class 'curator.exceptions.FailedExecution'>: Exception encountered. Rerun with loglevel DEBUG and/or check Elasticsearch logs for more information. Exception: ConnectionTimeout caused by - ReadTimeout(HTTPConnectionPool(host='127.0.0.1', port=9200): Read timed out. (read timeout=30))
May 04 00:43:35 logstash2026 systemd[1]: curator_actions_cluster_wide.service: Main process exited, code=exited, status=1/FAILURE
May 04 00:43:35 logstash2026 systemd[1]: curator_actions_cluster_wide.service: Failed with result 'exit-code'.
May 04 00:43:35 logstash2026 systemd[1]: Failed to start OpenSearch Curator action cluster_wide.
May 04 00:43:35 logstash2026 systemd[1]: curator_actions_cluster_wide.service: Consumed 12.964s CPU time.
```
The logs mention a timeout when trying to connect to OpenSearch on localhost, the unit is disabled. Output of `sudo systemctl status opensearch.service -l`:
```
● opensearch.service - OpenSearch$ sudo journalctl -e -n 100000 -u curator_actions_cluster_wide | grep 'Failed to complete action'
Loaded: loaded (/lib/systemd/system/opensearch.service;May 02 12:47:43 logstash1026 curator[2788572]: 2024-05-02 12:47:43,213 ERROR Failed to complete action: forcemerge. <class 'curator.exceptions.FailedExecution'>: Exception encountered. Rerun with loglevel DEBUG and/or check Elasticsearch logs for more information. Exception: ConnectionTimeout caused by - ReadTimeout(HTTPConnectionPool(host='127.0.0.1', disabled;port=9200): Read timed out. vendor preset: enabled(read timeout=21600))
Active: inactive (deadMay 09 12:18:08 logstash1026 curator[88855]: 2024-05-09 12:18:08,629 ERROR Failed to complete action: forcemerge. <class 'curator.exceptions.FailedExecution'>: Exception encountered. Rerun with loglevel DEBUG and/or check Elasticsearch logs for more information. Exception: ConnectionTimeout caused by - ReadTimeout(HTTPConnectionPool(host='127.0.0.1', port=9200): Read timed out. (read timeout=21600))
Docs: https://opensearch.org/
```
Relevant output of `top`:
```cwhite@logstash1026:~$ sudo journalctl -e -n 1000000 -u curator_actions_cluster_wide | grep 'Failed to complete action'
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMANDMay 02 12:47:43 logstash1026 curator[2788572]: 2024-05-02 12:47:43,213 ERROR Failed to complete action: forcemerge. <class 'curator.exceptions.FailedExecution'>: Exception encountered. Rerun with loglevel DEBUG and/or check Elasticsearch logs for more information. Exception: ConnectionTimeout caused by - ReadTimeout(HTTPConnectionPool(host='127.0.0.1', port=9200): Read timed out. (read timeout=21600))
May 09 12:18:08 logstash1026 curator[88855]: 2024-05-09 12:18:08,629 ERROR Failed to complete action: forcemerge. <class 'curator.exceptions.FailedExecution'>: Exception encountered. Rerun with loglevel DEBUG and/or check Elasticsearch logs for more information. 513981 opensea+ 20 0 1638.4g 37.0g 2.3g S 225.2 59.4 23120:46 javaException: ConnectionTimeout caused by - ReadTimeout(HTTPConnectionPool(host='127.0.0.1', port=9200): Read timed out. (read timeout=21600))
```