Maniphest T194835

mw2182 crash
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	jcrespo
	May 16 2018, 3:42 PM

Description

On boot, blocked on:

Alert!  System fatal error during previous boot
Cache and Core Box, Last Level Cache Error
Press F1 to continue, F2 for setup.

Logs:

2018-05-16T14:18:15-0500 CPU0000
CPU 1 has an internal error (IERR).

2018-05-16T14:18:14-0500 CPU9000
An OEM diagnostic event occurred.

2018-05-16T14:18:13-0500 CPU9000
An OEM diagnostic event occurred.

2018-05-16T14:18:13-0500 CPU0704
CPU 1 machine check error detected.

2018-05-16T14:17:39-0500 SYS1003
System CPU Resetting.

2018-05-16T14:17:11-0500 CPU9000
An OEM diagnostic event occurred.

2018-05-16T14:17:10-0500 CPU9000
An OEM diagnostic event occurred.

2018-05-16T14:17:09-0500 CPU9000
An OEM diagnostic event occurred.

2018-05-16T14:17:08-0500 CPU0704
CPU 1 machine check error detected.

Log Sequence Number: 140
Detailed Description:
System event log and OS logs may indicate that the exception is external to the processor.
Recommended Action:
1) Check system and operating system logs for exceptions. If no exceptions are found continue. 2) Turn system off and remove input power for one minute. Re-apply input power and turn system on. 3) Make sure the processor is seated correctly. 4) If the issue still persists, contact technical support. Refer to the product documentation to choose a convenient contact method.
Comment: root

Related Objects

Mentioned Here: T149287: Heating alerts for mw servers in eqiad

Event Timeline

jcrespo created this task.May 16 2018, 3:42 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 16 2018, 3:42 PM

I did a scap pull, but probably this need more hw research. Not removing/depooling it as no danger of problems for now on codfw.

jcrespo updated the task description. (Show Details)May 16 2018, 3:48 PM

Overheat? T149287

The server is out of warranty since January. @Papaul: Do we have any decommissioned servers from which we could swap the broken CPU?

MoritzMuehlenhoff triaged this task as Medium priority.May 28 2018, 11:25 AM

@MoritzMuehlenhoff yes we do have some decommissioned servers. This can be also a bad main board. What we can do first is to swap CPU position. Since the error is showing on CPU1 we can move CPU1 to CPU0 and CPU0 to CPU1 if we do have the error on CPU0 than the CPU1 is bad if we do have the error on CPU1 than we need to replace the main board.

Let me know if you have any questions.

In T194835#4238867, @Papaul wrote:

@MoritzMuehlenhoff yes we do have some decommissioned servers. This can be also a bad main board. What we can do first is to swap CPU position. Since the error is showing on CPU1 we can move CPU1 to CPU0 and CPU0 to CPU1 if we do have the error on CPU0 than the CPU1 is bad if we do have the error on CPU1 than we need to replace the main board.

Sounds good to me, let's do that! I have marked downtime for mw2182 and powered it off.

Papaul claimed this task.May 29 2018, 1:29 PM