Horizon console log for logging-logstash-04.logging.eqiad1.wikimedia.cloud shows only an OOM crash.

[1188573.398199] Out of memory: Killed process 7293 (java) total-vm:8170696kB, anon-rss:4772292kB, file-rss:0kB, shmem-rss:0kB, UID:496 pgtables:9908kB oom_score_adj:0

The instance is not responsive to ssh for me. I am going to try a "turn it off and on again" reset and see what happens. cc: @colewhite

Mentioned in SAL (#wikimedia-cloud) [2026-07-15T14:58:27Z] <bd808> Hard reboot of logging-logstash-04 via Horizon (T432254)

The reboot brought https://beta-logs.wmcloud.org/ back online. Ssh is working. And a manual curl call from deployment-deploy04 is successful. I expect the next scap run to get past this.

@colewhite There were some blips with this server earlier too (maybe a full 24 hours?) that seemed to resolve without intervention (or at least anything I could see in https://sal.toolforge.org/logging. I'm not sure if this is something you'd like to poke into or not to see if there is something that could be fixed or tuned.

Mentioned in SAL (#wikimedia-cloud) [2026-07-17T15:12:53Z] <bd808> Hard reboot of logging-logstash-04.logging.eqiad1.wikimedia.cloud; non-responsive to network (T432254)

@colewhite Can you give us some insight into what's going with logging-logstash-04.logging.eqiad1.wikimedia.cloud? It has failed twice this week.

Cole has been out of office this week. I expect he can start looking into things when he is back. There may just be some traffic change that is making a JVM sad. There may also be something going on in addition to the OOM at an OpenStack level: T432426: Network unavailable on a few VMs.

This is blowing up again right now:

[22:13:37] 22:13:37 The canary error rate checker failed: Generic connection error: HTTPConnectionPool(host='logs-api.svc.logging.eqiad1.wikimedia.cloud', port=9200): Max retries exceeded with url: /logstash-*/_search (Caused by NewConnectionError("HTTPConnection(host='logs-api.svc.logging.eqiad1.wikimedia.cloud', port=9200): Failed to establish a new connection: [Errno 101] Network is unreachable"))

logging-logstash-04 Instance Console Log via Horizon:

?2004hroot@logging-logstash-04:~# [   24.825426] cloud-init[1115]: Cloud-init v. 22.4.2 running 'modules:final' at Fri, 17 Jul 2026 19:53:14 +0000. Up 24.78 seconds.
[   24.902566] cloud-init[1115]: Cloud-init v. 22.4.2 finished at Fri, 17 Jul 2026 19:53:14 +0000. Datasource DataSourceOpenStackLocal [net,ver=2].  Up 24.89 seconds
[320204.667656] Out of memory: Killed process 6040 (java) total-vm:8184112kB, anon-rss:4834520kB, file-rss:0kB, shmem-rss:0kB, UID:496 pgtables:10132kB oom_score_adj:0
[368973.119119] Out of memory: Killed process 392696 (java) total-vm:8178888kB, anon-rss:4880224kB, file-rss:0kB, shmem-rss:0kB, UID:496 pgtables:10224kB oom_score_adj:0

Instance not responding to my ssh attempts.

bd808@mpb03:~$ ssh -vvv root@logging-logstash-04.logging.eqiad1.wikimedia.cloud
...
channel 0: open failed: connect failed: No route to host
debug2: channel_input_open_failure: channel 0: callback start
stdio forwarding failed
kex_exchange_identification: Connection closed by remote host
Connection closed by UNKNOWN port 65535

Mentioned in SAL (#wikimedia-cloud) [2026-07-22T22:24:24Z] <bd808> logging-logstash-04 hard reboot via Horizon (T432254)

Change #1314132 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] beta-logs: reduce Xmx available to logstash

https://gerrit.wikimedia.org/r/1314132

Change #1314132 merged by Cwhite:

[operations/puppet@production] beta-logs: reduce Xmx available to logstash

https://gerrit.wikimedia.org/r/1314132

Thanks @colewhite!