Maniphest T192902

Broken memory/CPU on mw1275
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	MoritzMuehlenhoff
	Apr 24 2018, 12:45 PM

Description

While reimaging mw1275 went down, "racadm getsel" lists a couple of errors, but I'm unsure if the CPU error is fallout from the broken DIMM or whether both are faulty?

The server is depooled, you can power if off for maintenance at any time.

Record:      2
Date/Time:   04/24/2018 09:53:04
Source:      system
Severity:    Non-Critical
Description: Correctable memory error rate exceeded for DIMM_B1.
-------------------------------------------------------------------------------
Record:      3
Date/Time:   04/24/2018 09:53:13
Source:      system
Severity:    Critical
Description: Correctable memory error rate exceeded for DIMM_B1.
-------------------------------------------------------------------------------
Record:      4
Date/Time:   04/24/2018 10:22:07
Source:      system
Severity:    Critical
Description: CPU 2 has an internal error (IERR).
-------------------------------------------------------------------------------
Record:      5
Date/Time:   04/24/2018 10:22:36
Source:      system
Severity:    Ok
Description: A problem was detected related to the previous server boot.
-------------------------------------------------------------------------------
Record:      6
Date/Time:   04/24/2018 10:22:36
Source:      system
Severity:    Critical
Description: Multi-bit memory errors detected on a memory device at location(s) DIMM_B1.
-------------------------------------------------------------------------------
Record:      7
Date/Time:   04/24/2018 12:18:26
Source:      system
Severity:    Critical
Description: Correctable memory error rate exceeded for DIMM_B1.

Event Timeline

MoritzMuehlenhoff created this task.Apr 24 2018, 12:45 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 24 2018, 12:45 PM

• Cmjohnson moved this task from Backlog to High Priority Task on the ops-eqiad board.May 2 2018, 2:34 PM

I swapped the DIMM with A1, cleared SEL and powered back on. Let's see if the error returns and/or moves.

@MoritzMuehlenhoff The error has not returned, go ahead and re-install. The error was correctable, so moving and reseating may have fixed the issue.

Thanks, I'll reimage it on Monday.

Mentioned in SAL (#wikimedia-operations) [2018-05-07T09:02:14Z] <moritzm> reimaging mw1275 (T192902)

The server has been reimaged and is currently serving production traffic without any issues, closing.

Broken memory/CPU on mw1275Closed, ResolvedPublicActions

Description

Event Timeline

Broken memory/CPU on mw1275
Closed, ResolvedPublic
Actions