Page MenuHomePhabricator
Create Task
Maniphest T258249

db1145 crashed - memory issues
Closed, ResolvedPublic
Actions

Description

db1145 rebooted itself:

Times in UTC

[09:50:33]  <+icinga-wm>	PROBLEM - Host db1145 is DOWN: PING CRITICAL - Packet loss = 100%

Startup message:

UEFI0079: One or more uncorrectable Memory errors occurred in the previous
boot.
Check the System Event Log (SEL) to identify the non-functional DIMM, and then
replace the DIMM.


Available Actions:
F1 to Continue and Retry Boot Order
F2 for System Setup (BIOS)
F10 for Lifecycle Controller
- Enable/Configure iDRAC
- Update or Backup/Restore Server Firmware
- Help Install an Operating System
F11 for Boot Manager

This host is a backup source and it is under warranty

Details

ProjectBranchLines +/-Subject
operations/puppetproduction+0 -1Revert "db1145: Disable notifications"
operations/puppetproduction+1 -0db1145: Disable notifications
Customize query in gerrit

Event Timeline

• Marostegui created this task.Jul 17 2020, 9:53 AM
Restricted Application added a project: SRE. · View Herald TranscriptJul 17 2020, 9:53 AM
• Marostegui added a comment.Jul 17 2020, 9:55 AM

More HW logs - we've got a broken DIMM apparently

Record:      1
Date/Time:   03/30/2020 23:27:03
Source:      system
Severity:    Ok
Description: Log cleared.
-------------------------------------------------------------------------------
Record:      2
Date/Time:   07/17/2020 09:48:30
Source:      system
Severity:    Critical
Description: The system memory has faced an uncorrectable multi-bit memory errors in the non-execution path of a memory device at the location DIMM_B2.
-------------------------------------------------------------------------------
Record:      3
Date/Time:   07/17/2020 09:48:30
Source:      system
Severity:    Ok
Description: An OEM diagnostic event occurred.
-------------------------------------------------------------------------------
Record:      4
Date/Time:   07/17/2020 09:48:30
Source:      system
Severity:    Ok
Description: An OEM diagnostic event occurred.
-------------------------------------------------------------------------------
Record:      5
Date/Time:   07/17/2020 09:48:30
Source:      system
Severity:    Ok
Description: An OEM diagnostic event occurred.
-------------------------------------------------------------------------------
Record:      6
Date/Time:   07/17/2020 09:48:31
Source:      system
Severity:    Critical
Description: Multi-bit memory errors detected on a memory device at location(s) DIMM_B2.
-------------------------------------------------------------------------------
Record:      7
Date/Time:   07/17/2020 09:48:39
Source:      system
Severity:    Critical
Description: The system memory has faced an uncorrectable multi-bit memory errors in the non-execution path of a memory device at the location DIMM_B2.
-------------------------------------------------------------------------------
Record:      8
Date/Time:   07/17/2020 09:48:39
Source:      system
Severity:    Ok
Description: An OEM diagnostic event occurred.
-------------------------------------------------------------------------------
Record:      9
Date/Time:   07/17/2020 09:48:40
Source:      system
Severity:    Ok
Description: An OEM diagnostic event occurred.
-------------------------------------------------------------------------------
Record:      10
Date/Time:   07/17/2020 09:48:40
Source:      system
Severity:    Ok
Description: An OEM diagnostic event occurred.
-------------------------------------------------------------------------------
Record:      11
Date/Time:   07/17/2020 09:48:47
Source:      system
Severity:    Critical
Description: The system memory has faced an uncorrectable multi-bit memory errors in the non-execution path of a memory device at the location DIMM_B2.
-------------------------------------------------------------------------------
Record:      12
Date/Time:   07/17/2020 09:48:47
Source:      system
Severity:    Ok
Description: An OEM diagnostic event occurred.
-------------------------------------------------------------------------------
Record:      13
Date/Time:   07/17/2020 09:48:47
Source:      system
Severity:    Ok
Description: An OEM diagnostic event occurred.
-------------------------------------------------------------------------------
Record:      14
Date/Time:   07/17/2020 09:48:48
Source:      system
Severity:    Ok
Description: An OEM diagnostic event occurred.
-------------------------------------------------------------------------------
Record:      15
Date/Time:   07/17/2020 09:48:48
Source:      system
Severity:    Critical
Description: CPU 2 machine check error detected.
-------------------------------------------------------------------------------
Record:      16
Date/Time:   07/17/2020 09:48:48
Source:      system
Severity:    Ok
Description: An OEM diagnostic event occurred.
-------------------------------------------------------------------------------
Record:      17
Date/Time:   07/17/2020 09:48:48
Source:      system
Severity:    Ok
Description: An OEM diagnostic event occurred.
-------------------------------------------------------------------------------
Record:      18
Date/Time:   07/17/2020 09:48:48
Source:      system
Severity:    Ok
Description: An OEM diagnostic event occurred.
-------------------------------------------------------------------------------
Record:      19
Date/Time:   07/17/2020 09:48:48
Source:      system
Severity:    Ok
Description: An OEM diagnostic event occurred.
-------------------------------------------------------------------------------
Record:      20
Date/Time:   07/17/2020 09:48:48
Source:      system
Severity:    Ok
Description: An OEM diagnostic event occurred.
-------------------------------------------------------------------------------
Record:      21
Date/Time:   07/17/2020 09:48:49
Source:      system
Severity:    Ok
Description: An OEM diagnostic event occurred.
-------------------------------------------------------------------------------
Record:      22
Date/Time:   07/17/2020 09:48:49
Source:      system
Severity:    Ok
Description: An OEM diagnostic event occurred.
-------------------------------------------------------------------------------
Record:      23
Date/Time:   07/17/2020 09:48:49
Source:      system
Severity:    Ok
Description: An OEM diagnostic event occurred.
-------------------------------------------------------------------------------
Record:      24
Date/Time:   07/17/2020 09:48:49
Source:      system
Severity:    Ok
Description: An OEM diagnostic event occurred.
-------------------------------------------------------------------------------
Record:      25
Date/Time:   07/17/2020 09:48:49
Source:      system
Severity:    Ok
Description: An OEM diagnostic event occurred.
-------------------------------------------------------------------------------
Record:      26
Date/Time:   07/17/2020 09:48:49
Source:      system
Severity:    Ok
Description: An OEM diagnostic event occurred.
-------------------------------------------------------------------------------
Record:      27
Date/Time:   07/17/2020 09:48:49
Source:      system
Severity:    Ok
Description: An OEM diagnostic event occurred.
-------------------------------------------------------------------------------
Record:      28
Date/Time:   07/17/2020 09:51:41
Source:      system
Severity:    Ok
Description: A problem was detected related to the previous server boot.
-------------------------------------------------------------------------------
Record:      29
Date/Time:   07/17/2020 09:51:41
Source:      system
Severity:    Critical
Description: Multi-bit memory errors detected on a memory device at location(s) DIMM_B2.
-------------------------------------------------------------------------------
gerritbot added a comment.Jul 17 2020, 9:55 AM

Change 613609 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] db1145: Disable notifications

https://gerrit.wikimedia.org/r/613609

gerritbot added a project: Patch-For-Review.Jul 17 2020, 9:55 AM

Change 613609 merged by Marostegui:
[operations/puppet@production] db1145: Disable notifications

https://gerrit.wikimedia.org/r/613609

RhinosF1 added a subscriber: RhinosF1.Jul 17 2020, 9:56 AM
• Marostegui added a comment.Jul 17 2020, 9:58 AM

I have hit F1, to continue its boot, so we can see the OS.
The memory on the OS looks ok:

root@db1145:~# free -g
              total        used        free      shared  buff/cache   available
Mem:            502           1         501           0           0         499
Swap:             7           0           7

I won't start MySQL until @jcrespo gives green light but we should probably get an RMA started for that particular DIMM @wiki_willy

• Marostegui triaged this task as Medium priority.Jul 17 2020, 9:58 AM
• Marostegui moved this task from Triage to In progress on the DBA board.
• Marostegui updated the task description. (Show Details)
Maintenance_bot removed a project: Patch-For-Review.Jul 17 2020, 10:10 AM
jcrespo added a comment.Jul 17 2020, 11:06 AM

I've started a restore process from yesterday's backups.

jcrespo assigned this task to wiki_willy.Jul 17 2020, 1:20 PM

The service is back up from backups so the backup service continues uninterrupted during the weekend. @wiki_willy let us know what is the next step as mentioned by Marostegui before and I can put down the server again next week if necessary.

jcrespo moved this task from In progress to Blocked external/Not db team on the DBA board.Jul 17 2020, 1:21 PM
• Marostegui added a comment.Jul 17 2020, 1:24 PM

@jcrespo remember I disabled notifications via puppet, I guess we should leave them disabled until the maintenance is done?

jcrespo added a comment.Jul 17 2020, 1:25 PM

Yes, I agree with that option. Thanks for creating the ticket and doing the initial triage!

• Marostegui added a comment.Jul 17 2020, 1:26 PM

Cool! <3

wiki_willy reassigned this task from wiki_willy to Jclark-ctr.Jul 17 2020, 5:09 PM
wiki_willy added a subscriber: Jclark-ctr.

@Jclark-ctr - can you check this one out when you're onsite next? It was only installed a few months ago, so we should be able to RMA the part pretty easily. Thanks, Willy

Cmjohnson added a subscriber: Cmjohnson.Jul 20 2020, 7:35 PM

TSR20200720182201_J2QF333.zip889 KBDownload
. @Jclark-ctr TSR report is attached

Jclark-ctr added a comment.Jul 20 2020, 10:15 PM

Confirmed: Service Request 1030121866 was successfully submitted.

Jclark-ctr added a comment.Jul 21 2020, 8:18 PM

@Marostegui Replacement Dimm has arrived please reach out to me for scheduling down time i am available for the next 2 hours but will be on site tomorrow 9:30am est

• Marostegui added a comment.Jul 22 2020, 5:00 AM

Thanks @Jclark-ctr - I am going to depool this hots so it is ready for when you arrive to the DC.

• Marostegui added a comment.Jul 22 2020, 5:08 AM

This being a backups source doesn't require depooling, but we need to check with @jcrespo when this host can be powered off.

Stashbot added a comment.Jul 22 2020, 7:40 AM

Mentioned in SAL (#wikimedia-operations) [2020-07-22T07:40:52Z] <jynus> stop db1145 for hw maintenance T258249

jcrespo added a comment.Jul 22 2020, 7:45 AM

Backups were taken from db1145 today and the host put down. Please ping here when maintenance is complete.

Jclark-ctr added a comment.Jul 22 2020, 2:21 PM

@jcrespo maintenance is completed

gerritbot added a comment.Jul 22 2020, 2:31 PM

Change 615430 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] Revert "db1145: Disable notifications"

https://gerrit.wikimedia.org/r/615430

gerritbot added a project: Patch-For-Review.Jul 22 2020, 2:31 PM

Change 615430 merged by Jcrespo:
[operations/puppet@production] Revert "db1145: Disable notifications"

https://gerrit.wikimedia.org/r/615430

jcrespo closed this task as Resolved.Jul 22 2020, 2:46 PM

Everything looking good. Thanks, @Jclark-ctr !

Maintenance_bot removed a project: Patch-For-Review.Jul 22 2020, 3:10 PM
Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL