Page MenuHomePhabricator
Create Task
Maniphest T240187

mw1280 crashed logging correctable memory errors
Closed, ResolvedPublic
Actions

Description

mw1280 crashed, mgmt console stuck, I had to powercycle it. The racadm getsel logged a ton of OEM errors ending up in Correctable memory error rate exceeded for DIMM_B1..

Worth to note that this host has a history of problems: T218006, T195734

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedDzahnT240187 mw1280 crashed logging correctable memory errors
Unknown Object (Task)

Event Timeline

elukey created this task.Dec 9 2019, 8:45 AM
Restricted Application added a project: SRE. · View Herald TranscriptDec 9 2019, 8:45 AM
wiki_willy assigned this task to Jclark-ctr.Dec 9 2019, 8:38 PM
wiki_willy subscribed.

Looks like it's out of warranty. We can purchase a replacement DIMM, but will need the correct specs to place the order.

Thanks
Willy

jcrespo moved this task from Backlog to Acknowledged on the SRE board.Dec 12 2019, 6:14 PM
wiki_willy moved this task from Backlog to Hardware Failure / Troubleshoot on the ops-eqiad board.Dec 28 2019, 9:05 PM
Jclark-ctr added a comment.Feb 19 2020, 9:11 PM

@wiki_willy going by dell support based on service tag
Part number: PR5D1
DIMM,32GB,2133,2RX4,8G,DDR4,R 2

wiki_willy added a subtask: Unknown Object (Task).Feb 19 2020, 9:23 PM
wiki_willy added a comment.Feb 19 2020, 9:25 PM

Created T245670 to have a replacement DIMM ordered and delivered.

Thanks,
Willy

MoritzMuehlenhoff subscribed.Feb 20 2020, 11:22 AM

The server went down with the following error today:

Record:      216
Date/Time:   02/20/2020 11:16:09
Source:      system
Severity:    Critical
Description: Correctable memory error rate exceeded for DIMM_B1.

I've depooled it until the DIMM module has arrived/been swapped.

jijiki added a project: serviceops.Feb 24 2020, 11:07 AM
Volans subscribed.Feb 27 2020, 11:57 AM

The host has been down a week, hence it has been removed from PuppetDB and the Netbox report catched it.
Updated Netbox setting it's state to Failed. Please follow https://wikitech.wikimedia.org/wiki/Server_Lifecycle#Active_-%3E_Failed

Jclark-ctr closed subtask Unknown Object (Task) as Resolved.Mar 17 2020, 4:29 PM

Replacement Dimm has arrived

Dzahn subscribed.Mar 17 2020, 4:33 PM

@Jclark-ctr The server is depooled, you can do the replacement any time.

Dzahn triaged this task as Medium priority.Mar 17 2020, 4:33 PM
Jclark-ctr added a comment.Mar 17 2020, 4:34 PM

Thanks taking care of now! @Dzahn

Jclark-ctr added a comment.Mar 17 2020, 4:41 PM

Replaced Failed drive host booting now

Dzahn claimed this task.Mar 17 2020, 4:45 PM
Dzahn added a subscriber: Jclark-ctr.

Thanks @Jclark-ctr ! I could get it per SSH now. I'll take it to get it back into production, if you are done.

Stashbot added a comment.Mar 17 2020, 4:46 PM

Mentioned in SAL (#wikimedia-operations) [2020-03-17T16:46:53Z] <mutante> mw1280 back after long downtime due to broken RAM, added back into puppet (T240187)

Dzahn added a comment.Mar 17 2020, 5:15 PM

after puppet runs host was added back in Icinga.

then: CRITICAL: 944 mismatched wikiversions

after a looong scap pull it is all green now

https://icinga.wikimedia.org/cgi-bin/icinga/status.cgi?host=mw1280&scroll=220

will pool again

Dzahn closed this task as Resolved.Mar 17 2020, 5:19 PM

17:18 <+logmsgbot> !log dzahn@cumin1001 conftool action : set/pooled=yes; selector: name=mw1280.eqiad.wmnet

elukey mentioned this in T251077: mw1280 correctable memory errors logged in getsel.Apr 27 2020, 6:26 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL