Page MenuHomePhabricator
Create Task
Maniphest T188595

Memory test failure on elastic1021
Closed, ResolvedPublic
Actions

Description

While rebooting the elasticsearch eqiad cluster, elastic1021 failed to come back up with the following message:

Error: Memory initialization warning detected.
MEMBIST Memory Test failure DIMM A3

@Cmjohnson: could you have a look (and probably change that RAM)?

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedRobHT188595 Memory test failure on elastic1021
Unknown Object (Task)
 Resolved• CmjohnsonT189727 decommission elastic1021

Event Timeline

Gehel created this task.Mar 1 2018, 8:04 AM
Restricted Application added a project: SRE. · View Herald TranscriptMar 1 2018, 8:04 AM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Stashbot subscribed.Mar 1 2018, 8:20 AM

Mentioned in SAL (#wikimedia-operations) [2018-03-01T08:20:48Z] <gehel> banning elastic1021 from cluster (failed memory) - T188595

MoritzMuehlenhoff triaged this task as Medium priority.Mar 1 2018, 9:32 AM
• Cmjohnson added a comment.Mar 6 2018, 3:32 PM

Swapped DIMM A3 to B2 to see if the error follows the DIMM. Powered on

• Cmjohnson moved this task from Backlog to High Priority Task on the ops-eqiad board.Mar 6 2018, 3:33 PM
• Cmjohnson added a comment.Mar 8 2018, 4:30 PM

The error did follow the DIMM, The server is out of warranty but I will see if I can snag a similar DIMM from a decommissioned server.

• Cmjohnson added subscribers: RobH, faidon.Mar 8 2018, 4:36 PM

I do not have anything that size as a spare. @RobH and @faidon will need to comment on buying a new DIMM.

Current DIMM is Samsung 16GB 2Rx4PC3L-12800R-11-12-E2-D4

RobH mentioned this in Unknown Object (Task).Mar 8 2018, 4:59 PM
Gehel moved this task from Incoming to Needs Reporting on the Discovery-Search (Current work) board.Mar 13 2018, 5:37 PM

The decision is to not replace this out of warranty RAM. We'll run with 3% less capacity until this batch of servers is renewed (in ~ 1year).

• Cmjohnson added a comment.Mar 14 2018, 7:09 PM

@Gehel do you want to decommission this server then?

• Cmjohnson moved this task from High Priority Task to Blocked on the ops-eqiad board.Mar 14 2018, 7:10 PM
RobH closed subtask Unknown Object (Task) as Declined.Mar 14 2018, 7:11 PM
RobH mentioned this in T189727: decommission elastic1021.
RobH closed this task as Resolved.Mar 14 2018, 7:17 PM
RobH claimed this task.

Discussion on T189223 resulted in the decision to decommission elastic1021. I've created T189727 listing the decom process/steps. I'm going to close this task as resolved, since we're decommissioning the host.

• Cmjohnson closed subtask T189727: decommission elastic1021 as Resolved.Apr 18 2018, 4:24 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits