Page MenuHomePhabricator
Create Task
Maniphest T427376

db2189 crashed
Closed, ResolvedPublic
Actions

Description

db2189 went down and HW logs only show multiple entries of:

-------------------------------------------------------------------------------
Record:      98
Date/Time:   05/26/2026 11:51:03
Source:      system
Severity:    Ok
Description: An OEM diagnostic event occurred.
-------------------------------------------------------------------------------
Record:      99
Date/Time:   05/26/2026 11:51:03
Source:      system
Severity:    Ok
Description: An OEM diagnostic event occurred.
-------------------------------------------------------------------------------

Is there anything that can be done on-site to try to troubleshoot this further and check if there's something else failing?
Thanks!

Related Objects
Search...

StatusSubtypeAssignedTask
 OpenNoneT426720 May 2026 SRE reboots
Restricted Task
Restricted Task
 ResolvedJhancock.wmT427376 db2189 crashed

Event Timeline

Marostegui created this task.Wed, May 27, 11:12 AM
Restricted Application added a project: DC-Ops. · View Herald TranscriptWed, May 27, 11:12 AM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Marostegui triaged this task as Medium priority.Wed, May 27, 11:12 AM
Marostegui moved this task from Triage to In progress on the DBA board.
Marostegui added a parent task: Restricted Task.
FCeratto-WMF mentioned this in T427377: sre.mysql.pool: remove downtime before pooling.Wed, May 27, 11:21 AM
Maintenance_bot added a project: SRE.Wed, May 27, 11:29 AM
Jhancock.wm moved this task from Backlog to Hardware Failure / Troubleshoot on the ops-codfw board.Wed, May 27, 2:22 PM
Jhancock.wm subscribed.Wed, May 27, 2:34 PM

working on it. might reboot a few times.

Stashbot added a comment.Wed, May 27, 2:38 PM

Mentioned in SAL (#wikimedia-operations) [2026-05-27T14:38:57Z] <fceratto@cumin1003> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 99 days, 0:00:00 on db2189.codfw.wmnet with reason: crashed T427376

FCeratto-WMF subscribed.Wed, May 27, 2:38 PM

(added a long downtime just in case)

Jhancock.wm added a comment.Wed, May 27, 3:22 PM

@FCeratto-WMF okay the error code we got was inconclusive. it could mean a lot of things including just out of date firmware. I've updated the bios and the idrac. I do see a cpu machine check error but it should have also been resolved by the firmware update. should be good to add this one back now.

Jhancock.wm claimed this task.Wed, May 27, 3:23 PM
Marostegui added a comment.Thu, May 28, 5:21 AM
In T427376#11960059, @Jhancock.wm wrote:

@FCeratto-WMF okay the error code we got was inconclusive. it could mean a lot of things including just out of date firmware. I've updated the bios and the idrac. I do see a cpu machine check error but it should have also been resolved by the firmware update. should be good to add this one back now.

Thanks Jenn - I will repool the host.

Marostegui closed this task as Resolved.Thu, May 28, 7:38 AM
Maintenance_bot moved this task from In progress to Done on the DBA board.Thu, May 28, 8:29 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits