db1155 HW memory errors
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Marostegui
	May 19 2025, 5:27 AM

Description

[9049296.216488] mce: Uncorrected hardware memory error in user-access at 506a064900
[9049296.216496] mce: [Hardware Error]: Machine check events logged
[9049296.219992] Memory failure: 0x506a064: Sending SIGBUS to mysqld:4072 due to hardware memory corruption
[9049296.233458] Memory failure: 0x506a064: recovery action for dirty LRU page: Recovered
[9049307.944609] MCE: Killing mysqld:4072 due to hardware memory corruption fault at 7fcfec064904

Record:      80
Date/Time:   05/19/2025 06:14:34
Source:      system
Severity:    Critical
Description: Multi-bit memory errors detected on a memory device at location(s) DIMM_B5.
-------------------------------------------------------------------------------

Details

Related Changes in Gerrit:

	Subject	Repo	Branch	Lines +/-
	db1155: Disable notifications	operations/puppet	production	+1 -0

Customize query in gerrit

Related Objects

Mentioned In: T394859: Quarry WMCloud (ruwiki_p, section s6) experiencing sustained replication lag (~16 h)
T394653: Test MariaDB 10.11.13

Event Timeline

Marostegui created this task.May 19 2025, 5:27 AM

Restricted Application added a project: DC-Ops. · View Herald TranscriptMay 19 2025, 5:27 AM

Any chances we can replace this DIMM with another one from a decommissioned server?

Maintenance_bot added a project: SRE.May 19 2025, 5:29 AM

VRiley-WMF claimed this task.May 19 2025, 5:33 PM

VRiley-WMF moved this task from Backlog to Hardware Failure / Troubleshoot on the ops-eqiad board.

@VRiley-WMF I'd need to depool the server first.

Hey @Marostegui we certainly do! Is there a preferred time or date to swap out the memory? Let us know, thanks!

@VRiley-WMF thanks! I'll have the host ready for you tomorrow if that's ok?

@Marostegui That works for me. I'll plan for it then. Thanks!

Change #1147921 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] db1155: Disable notifications

https://gerrit.wikimedia.org/r/1147921

gerritbot added a project: Patch-For-Review.May 20 2025, 4:49 AM

Mentioned in SAL (#wikimedia-operations) [2025-05-20T04:51:22Z] <marostegui> Stop mariadb on db1155, wiki replicas will show lag on: s2, s4, s6 and s7 T394624

Change #1147921 merged by Marostegui:

[operations/puppet@production] db1155: Disable notifications

https://gerrit.wikimedia.org/r/1147921

@VRiley-WMF db1155 is now off and ready for you to replace the memory whenever you want.

Maintenance_bot removed a project: Patch-For-Review.May 20 2025, 5:30 AM

Marostegui mentioned this in T394653: Test MariaDB 10.11.13.May 21 2025, 6:43 AM

GPSLeo subscribed.May 21 2025, 6:46 AM

taavi mentioned this in T394859: Quarry WMCloud (ruwiki_p, section s6) experiencing sustained replication lag (~16 h).May 21 2025, 7:54 AM

taavi subscribed.May 21 2025, 8:54 AM

When is this expected to be solved? Because of this problem many important maintenance and monitoring tools are broken. This should have UBN priority.

fnegri subscribed.May 21 2025, 2:45 PM

In T394624#10844404, @GPSLeo wrote:

When is this expected to be solved? Because of this problem many important maintenance and monitoring tools are broken. This should have UBN priority.

Please note that we are working on it as you can see above. Also, keep in mind that wiki replicas infra isn't considered mission critical environment, and lag can happen any time, while it is not ideal, it can happen (https://wikitech.wikimedia.org/wiki/Help:Wiki_Replicas#Identifying_lag). Luckily, we've been running this infra for years with pretty much no outages, but these things can happen and we work on the when we can. Generally they get resolved faster, but again, this is not a production critical environment.

Taking this unit down for the memory swap.

This is completed

In T394624#10845303, @VRiley-WMF wrote:

This is completed

Thanks!

I started the mariadb deamons.

Started replication.

Maintenance_bot moved this task from In progress to Done on the DBA board.May 21 2025, 6:29 PM

db1155 HW memory errorsClosed, ResolvedPublicActions

Description

Details

Related Objects

Event Timeline

db1155 HW memory errors
Closed, ResolvedPublic
Actions