Page MenuHomePhabricator
Create Task
Maniphest T429113

es2045 down
Closed, ResolvedPublic
Actions

Description

es2045 went down and these are the HW errors:

-------------------------------------------------------------------------------
Record:      51
Date/Time:   06/14/2026 05:16:32
Source:      system
Severity:    Ok
Description: A problem was detected related to the previous server boot.
-------------------------------------------------------------------------------
Record:      52
Date/Time:   06/14/2026 05:16:32
Source:      system
Severity:    Critical
Description: CPU 1 machine check error detected.
-------------------------------------------------------------------------------
Record:      53
Date/Time:   06/14/2026 05:16:32
Source:      system
Severity:    Ok
Description: An OEM diagnostic event occurred.
-------------------------------------------------------------------------------
Record:      54
Date/Time:   06/14/2026 05:16:33
Source:      system
Severity:    Ok
Description: An OEM diagnostic event occurred.
-------------------------------------------------------------------------------
Record:      55
Date/Time:   06/14/2026 05:16:33
Source:      system
Severity:    Ok
Description: An OEM diagnostic event occurred.
-------------------------------------------------------------------------------
Record:      56
Date/Time:   06/14/2026 05:16:33
Source:      system
Severity:    Ok
Description: An OEM diagnostic event occurred.
-------------------------------------------------------------------------------
Record:      57
Date/Time:   06/14/2026 05:16:33
Source:      system
Severity:    Ok
Description: An OEM diagnostic event occurred.
-------------------------------------------------------------------------------
Record:      58
Date/Time:   06/14/2026 05:16:33
Source:      system
Severity:    Ok
Description: An OEM diagnostic event occurred.
-------------------------------------------------------------------------------
Record:      59
Date/Time:   06/14/2026 05:16:33
Source:      system
Severity:    Ok
Description: An OEM diagnostic event occurred.
-------------------------------------------------------------------------------
Record:      60
Date/Time:   06/14/2026 05:16:33
Source:      system
Severity:    Ok
Description: An OEM diagnostic event occurred.
-------------------------------------------------------------------------------
Record:      61
Date/Time:   06/14/2026 05:16:33
Source:      system
Severity:    Ok
Description: An OEM diagnostic event occurred.
-------------------------------------------------------------------------------
Record:      62
Date/Time:   06/14/2026 05:16:34
Source:      system
Severity:    Ok
Description: An OEM diagnostic event occurred.
-------------------------------------------------------------------------------
Record:      63
Date/Time:   06/14/2026 05:16:34
Source:      system
Severity:    Ok
Description: An OEM diagnostic event occurred.
-------------------------------------------------------------------------------
Record:      64
Date/Time:   06/14/2026 05:16:34
Source:      system
Severity:    Ok
Description: An OEM diagnostic event occurred.
-------------------------------------------------------------------------------
Record:      65
Date/Time:   06/14/2026 05:16:34
Source:      system
Severity:    Ok
Description: An OEM diagnostic event occurred.
-------------------------------------------------------------------------------
Record:      66
Date/Time:   06/14/2026 05:16:34
Source:      system
Severity:    Ok
Description: An OEM diagnostic event occurred.
-------------------------------------------------------------------------------
Record:      67
Date/Time:   06/14/2026 05:16:34
Source:      system
Severity:    Ok
Description: An OEM diagnostic event occurred.
-------------------------------------------------------------------------------
Record:      68
Date/Time:   06/14/2026 05:16:34
Source:      system
Severity:    Ok
Description: An OEM diagnostic event occurred.
-------------------------------------------------------------------------------
Record:      69
Date/Time:   06/14/2026 05:16:34
Source:      system
Severity:    Ok
Description: An OEM diagnostic event occurred.
-------------------------------------------------------------------------------
Record:      70
Date/Time:   06/14/2026 05:16:35
Source:      system
Severity:    Ok
Description: An OEM diagnostic event occurred.
-------------------------------------------------------------------------------
Record:      71
Date/Time:   06/14/2026 05:16:35
Source:      system
Severity:    Ok
Description: An OEM diagnostic event occurred.
-------------------------------------------------------------------------------
Record:      72
Date/Time:   06/14/2026 05:16:35
Source:      system
Severity:    Ok
Description: An OEM diagnostic event occurred.
-------------------------------------------------------------------------------
Record:      73
Date/Time:   06/14/2026 05:16:35
Source:      system
Severity:    Ok
Description: An OEM diagnostic event occurred.
-------------------------------------------------------------------------------
Record:      74
Date/Time:   06/14/2026 05:16:35
Source:      system
Severity:    Ok
Description: An OEM diagnostic event occurred.
-------------------------------------------------------------------------------
Record:      75
Date/Time:   06/14/2026 05:16:35
Source:      system
Severity:    Ok
Description: An OEM diagnostic event occurred.
-------------------------------------------------------------------------------
Record:      76
Date/Time:   06/14/2026 05:16:35
Source:      system
Severity:    Ok
Description: An OEM diagnostic event occurred.
-------------------------------------------------------------------------------
Record:      77
Date/Time:   06/14/2026 05:16:35
Source:      system
Severity:    Ok
Description: An OEM diagnostic event occurred.
-------------------------------------------------------------------------------
Record:      78
Date/Time:   06/14/2026 05:16:36
Source:      system
Severity:    Ok
Description: An OEM diagnostic event occurred.
-------------------------------------------------------------------------------
Record:      79
Date/Time:   06/14/2026 05:16:36
Source:      system
Severity:    Ok
Description: An OEM diagnostic event occurred.
-------------------------------------------------------------------------------
Record:      80
Date/Time:   06/14/2026 05:16:36
Source:      system
Severity:    Ok
Description: An OEM diagnostic event occurred.
-------------------------------------------------------------------------------
Record:      81
Date/Time:   06/14/2026 05:16:36
Source:      system
Severity:    Ok
Description: An OEM diagnostic event occurred.
-------------------------------------------------------------------------------
Record:      82
Date/Time:   06/14/2026 05:16:36
Source:      system
Severity:    Ok
Description: An OEM diagnostic event occurred.
-------------------------------------------------------------------------------
Record:      83
Date/Time:   06/14/2026 05:16:36
Source:      system
Severity:    Ok
Description: An OEM diagnostic event occurred.
-------------------------------------------------------------------------------
Record:      84
Date/Time:   06/14/2026 05:16:36
Source:      system
Severity:    Ok
Description: An OEM diagnostic event occurred.
-------------------------------------------------------------------------------

Related Objects
Search...

Event Timeline

Marostegui created this task.Jun 15 2026, 5:34 AM
Restricted Application removed a project: Patch-For-Review. · View Herald TranscriptJun 15 2026, 5:34 AM
Marostegui removed Marostegui as the assignee of this task.Jun 15 2026, 5:34 AM
Marostegui triaged this task as Medium priority.
Marostegui added a project: ops-codfw.
Restricted Application added a project: DC-Ops. · View Herald TranscriptJun 15 2026, 5:34 AM
Maintenance_bot added a project: SRE.Jun 15 2026, 6:29 AM
Jhancock.wm moved this task from Backlog to Hardware Failure / Troubleshoot on the ops-codfw board.Jun 15 2026, 2:57 PM
Jhancock.wm claimed this task.Jun 15 2026, 3:25 PM
Jhancock.wm subscribed.

@Marostegui did a cold reboot and BIOS update. did an idrac update too to get better data later. Looks like it resolved the issue, but if it happens again, let us know. the server is less than 2 years old and we should be able to get a replacement part from Dell. but it would need to happen again to justify it.

Jhancock.wm added a comment.EditedJun 15 2026, 3:32 PM

i did find this from when we first got the server. same error and i think it was the same or similar process that preceded it. T381549

Marostegui added a comment.Jun 15 2026, 3:56 PM
In T429113#12020088, @Jhancock.wm wrote:

i did find this from when we first got the server. same error and i think it was the same or similar process that preceded it. T381549

would you talk to dell about it or you want me to repool?

Jhancock.wm added a comment.Jun 15 2026, 5:00 PM

i dug a little more into the errors. it's most likely a firmware issue than anything else. The error doesn't usually indicate that the CPU has an error, but that it caught an error in a process. (though it could still be the CPU. it's honestly a terrible error code.) I think it's fine for now. with the firmware update, it should give a more precise error code if it happens again.

in short, repool for now. we'll either never see it again or we'll get a more definite clue to what is wrong.

Marostegui added a comment.Jun 16 2026, 5:42 AM

Thanks @Jhancock.wm I will start the cloning and repooling. Thank you!

Jhancock.wm closed this task as Resolved.Jun 16 2026, 2:03 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits