@Papaul while this is not an emergency, we're already missing a bunch of servers, so ping me if there is anything I can do to help move this forward.

@Gehel Please power the server off if it is not off. Thanks.

I've just powered down elastic2038 via the mgmt interface.

The server warranty is until Nov 2021 and the firmware on the serve is old. If i call DELL they will ask me first to upgrade the firmware.
Old version
BIOS Version 1.5.6
Lifecycle Controller Firmware 3.21.21.21

Clear log before upgrade.
New version
BIOS Version 1.7.0
Lifecycle Controller Firmware 3.21.26.22

Firmware upgrade.
The next step will be to monitor the system and see if we have the same error. IF we do I will swap DIMM A2 with DIMM B2 if the error follows (shows in DIMM B2) I will then call DELL for memory replacement if the error stills show on DIMM A2 it will be bad slot on the mainboard.

I power the system back up.

@Gehel Since yesterday there is no error reported in the log. Can you repool the server so we can monitor it while it is under load.

Thanks.

@Papaul server has been repooled since it restarted. Let's blame cosmic rays until we prove otherwise?

@Gehel It crashed today. Just went down and had no console output. Then i powercycled it.. Then it was back.

Note the service came back before the host was back.. so it did fail-over succesfully:

20:30 <+icinga-wm> PROBLEM - Host elastic2038 is DOWN: PING CRITICAL - Packet loss = 100%
20:31 <+icinga-wm> PROBLEM - ElasticSearch health check for shards on 9443 on search.svc.codfw.wmnet is CRITICAL: CRITICAL - elasticsearch https://search.svc.codfw.wmnet:9443/_cluster/health error while 
                   fetching: HTTPSConnectionPool(host=search.svc.codfw.wmnet, port=9443): Read timed out. (read timeout=4) https://wikitech.wikimedia.org/wiki/Search%23Administration
20:32 < mutante> !log powercycling elastic2038

20:33 <+icinga-wm> RECOVERY - ElasticSearch health check for shards on 9443 on search.svc.codfw.wmnet is OK: OK - elasticsearch status production-search-omega-codfw: cluster_name: production-search-omega-codfw, 
                   number_of_nodes: 14, active_shards: 3158, unassigned_shards: 225, active_primary_shards: 1128, initializing_shards: 0, delayed_unassigned_shards: 225, status: yellow, number_of_data_nodes: 
                   14, number_of_pending_tasks: 0, relocating_shards: 0

20:34 <+icinga-wm> RECOVERY - Host elastic2038 is UP: PING OK - Packet loss = 0%, RTA = 36.10 ms

Mentioned in SAL (#wikimedia-operations) [2019-05-03T09:20:59Z] <gehel> ban elastic2038 from elastic clusters pending memory issue investigation - T217398

It looks like we need to investigate this a bit more

In T217398#4999270, @Papaul wrote:

The next step will be to monitor the system and see if we have the same error. IF we do I will swap DIMM A2 with DIMM B2 if the error follows (shows in DIMM B2) I will then call DELL for memory replacement if the error stills show on DIMM A2 it will be bad slot on the mainboard.

@Papaul: it looks like it is time to switch those DIMMs.

elastic2038 is banned from the cluster, and downtimed in icinga. Feel free to do whatever you need with it.

@Gehel DIMM swap complete

it's fully down now:

https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=1&host=elastic2038

It looks like the error is showing now on DIMM B2, so we have a bad DIMM. I will go ahead and request a replacement.

Description Date and Time
Correctable memory error rate exceeded for DIMM_B2. Fri 10 May 2019 16:49:06
A problem was detected related to the previous server boot. Fri 10 May 2019 09:40:39
Multi-bit memory errors detected on a memory device at location(s) DIMM_B2. Fri 10 May 2019 09:40:39
An OEM diagnostic event occurred. Fri 10 May 2019 09:38:16
An OEM diagnostic event occurred. Fri 10 May 2019 09:38:16
An OEM diagnostic event occurred. Fri 10 May 2019 09:38:16
An OEM diagnostic event occurred. Fri 10 May 2019 09:38:16

Create Dispatch: Success
You have successfully submitted request SR990577292.

Mentioned in SAL (#wikimedia-operations) [2019-05-14T19:25:43Z] <gehel> ban elastic2038 from elasticsearch cluster for memory replacement - T217398

Mentioned in SAL (#wikimedia-operations) [2019-05-14T19:28:11Z] <gehel> shutting down elastic2038 for memory replacement - T217398

Started up just now, forgot to add the task in the SAL :)

Mentioned in SAL (#wikimedia-operations) [2019-05-15T17:19:32Z] <onimisionipe> unban elastic2038 from shard allocation - T217398

Mentioned in SAL (#wikimedia-operations) [2019-05-16T02:27:16Z] <onimisionipe> pooling elastic2038 after unbanning - T217398