mw2444 down
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	colewhite
	Sep 7 2023, 8:52 PM

Project Tags

Referenced Files

None

Subscribers

Description

Icinga indicates host is hard down as of 2023-09-07 20:00:27.

SEL:

-------------------------------------------------------------------------------
Record:      18
Date/Time:   09/07/2023 20:00:27
Source:      system
Severity:    Ok
Description: A problem was detected related to the previous server boot.
-------------------------------------------------------------------------------
Record:      19
Date/Time:   09/07/2023 20:00:27
Source:      system
Severity:    Ok
Description: An OEM diagnostic event occurred.
-------------------------------------------------------------------------------

[same description]

Record:      32
Date/Time:   09/07/2023 20:00:28
Source:      system
Severity:    Ok
Description: An OEM diagnostic event occurred.
-------------------------------------------------------------------------------
Record:      33
Date/Time:   09/07/2023 20:00:28
Source:      system
Severity:    Critical
Description: CPU 2 machine check error detected.
-------------------------------------------------------------------------------
Record:      34
Date/Time:   09/07/2023 20:00:29
Source:      system
Severity:    Ok
Description: An OEM diagnostic event occurred.
-------------------------------------------------------------------------------

[same description]

Record:      53
Date/Time:   09/07/2023 20:00:31
Source:      system
Severity:    Ok
Description: An OEM diagnostic event occurred.
-------------------------------------------------------------------------------

Event Timeline

colewhite created this task.Sep 7 2023, 8:52 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 7 2023, 8:52 PM

colewhite updated the task description. (Show Details)Sep 7 2023, 8:56 PM

colewhite edited projects, added: serviceops-radar; removed: serviceops-deprecated.Sep 7 2023, 9:02 PM

Maintenance_bot added a project: SRE.Sep 7 2023, 9:29 PM

Jhancock.wm moved this task from Backlog to Hardware Failure / Troubleshoot on the ops-codfw board.Sep 8 2023, 12:09 PM

error found in the lifecycle log

CPU 2 machine check error detected.

I powered down the server and drained the flea power. waited 5 minutes. server is back up and pinging at the moment but I will leave this open to see if the error fires again over the weekend. keeping my fingies crossed.

@Jhancock.wm I am afraid the server is dead again :(

jijiki edited projects, added: serviceops-deprecated; removed: serviceops-radar.Sep 11 2023, 9:21 AM

I opened a Dell support ticket to get a replacement. I've rebooted it for now but expect it to go down again.
SR: 175669963

I've replaced the CPU. We should know by Friday if it has issues. I will leave the ticket open until then.

return tracking for the bad part (which I will also hold until Friday): 783629071254

I'm putting mw2444 back into pooled=no (instead of pooled=inactive) so it gets scap updates and stops warning, however I'll wait until we're sure it's stable before actually putting it back in production.

Mentioned in SAL (#wikimedia-operations) [2023-09-13T10:11:06Z] <claime> set/pooled=no; selector: name=mw2444.codfw.wmnet - T345884

@Clement_Goubert this remained stable for a good while. You should be able to retool it now. Thank you for humoring my wait time.

Mentioned in SAL (#wikimedia-operations) [2023-09-15T14:05:42Z] <claime> repooling mw2444.codfw.wmnet - T345884

Repooled, thank you!

mw2444 downClosed, ResolvedPublicActions

Description

Event Timeline

mw2444 down
Closed, ResolvedPublic
Actions