Page MenuHomePhabricator

hw troubleshooting: DIMM failure for mc2040.codfw.wmnet
Closed, ResolvedPublicRequest

Description

  • - Provide FQDN of system.
  • - If other than a hard drive issue, please depool the machine (and confirm that it’s been depooled) for us to work on it. If not, please provide time frame for us to take the machine down.
  • - Put system into a failed state in Netbox.
  • - Provide urgency of request, along with justification (redundancy, dependencies, etc)
  • - Describe issue and/or attach hardware failure log. (Refer to https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook if you need help)
  • - Assign correct project tag and appropriate owner (based on above). Also, please ensure the service owners of the host(s) are added as subscribers to provide any additional input.

FQDN: mc2040.codfw.wmnet
Urgency: medium
Issue: DIMM A2 errors

-------------------------------------------------------------------------------
Record:      20
Date/Time:   09/05/2023 19:48:03
Source:      system
Severity:    Critical
Description: Multi-bit memory errors detected on a memory device at location(s) DIMM_A2.
-------------------------------------------------------------------------------
Record:      21
Date/Time:   09/05/2023 19:50:03
Source:      system
Severity:    Ok
Description: A problem was detected related to the previous server boot.
-------------------------------------------------------------------------------
Record:      22
Date/Time:   09/05/2023 19:50:03
Source:      system
Severity:    Ok
Description: The self-heal operation successfully completed at DIMM DIMM_A2.
-------------------------------------------------------------------------------
Record:      23
Date/Time:   09/05/2023 19:50:03
Source:      system
Severity:    Ok
Description: The self-heal operation successfully completed at DIMM DIMM_A2.
-------------------------------------------------------------------------------
Record:      24
Date/Time:   09/05/2023 19:50:03
Source:      system
Severity:    Critical
Description: Multi-bit memory errors detected on a memory device at location(s) DIMM_A2.
-------------------------------------------------------------------------------

Thanks

Event Timeline

Mentioned in SAL (#wikimedia-operations) [2023-09-07T08:22:59Z] <cgoubert@cumin1001> START - Cookbook sre.hosts.downtime for 7 days, 0:00:00 on mc2040.codfw.wmnet with reason: T345802 - hw troubleshooting

Mentioned in SAL (#wikimedia-operations) [2023-09-07T08:23:12Z] <cgoubert@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on mc2040.codfw.wmnet with reason: T345802 - hw troubleshooting

Dear @Jhancock.wm or @Papaul, server can be shutdown and checked at your convenience, this part of the stack has failovers in place. Thank you!

I reseated DIMM_A2. I also found this ticket from January T326834. In that one B2 was having errors and I moved it to A2. Now that A2 is having issues, it's fair to assume that it's the problem.

This server is still under warranty so I will be opening a case with Dell to get it replaced.

@Clement_Goubert the defective DIMM has been replaced and booted up. Error hasn't repeated yet.

The self-heal operation successfully completed at DIMM DIMM_A2.

Going to close the ticket for now. If the error comes back this week, please reopen.

I've set the Netbox status back to Active.