Page MenuHomePhabricator

hw troubleshooting: Memory failure for cp6008.drmrs.wmnet
Open, MediumPublicRequest

Description

This is not urgent as we have redundant systems.

SEL says:

Multi-bit memory errors detected on a memory device at location(s) DIMM_B5.

The system is stuck at:

Screenshot at 2026-07-08 20-58-47.png (981×452 px, 59 KB)

It's been depooled and is ready for service.

  • - Provide FQDN of system.
  • - If other than a hard drive issue, please depool the machine (and confirm that it’s been depooled) for us to work on it. If not, please provide time frame for us to take the machine down.
  • - Put system into a failed state in Netbox.
  • - Provide urgency of request, along with justification (redundancy, dependencies, etc)
  • - Describe issue and/or attach hardware failure log. (Refer to https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook if you need help)
  • - Assign correct project tag and appropriate owner (based on above). Also, please ensure the service owners of the host(s) are added as subscribers to provide any additional input.

Event Timeline

Mentioned in SAL (#wikimedia-operations) [2026-07-09T04:09:59Z] <brett@cumin2002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 14 days, 0:00:00 on cp6008.drmrs.wmnet with reason: Hardware failure - T431651

Hi DC-Ops. Just following up on this in case it was missed, thanks! This is not urgent but will be good to know how long it may take.

My understanding of the situation:

  • We don't have any spare DIMMs at the edge, so we're unable to install any new sticks
  • drmrs will be the first to be refreshed with our new hardware layout, so we could simply keep this server out of service until the refresh
  • We could also simply remove the DIMM and repool the host: drmrs hosts have 12 32GB sticks, so we'd lose either 32GB or 64GB (depending on whether Dell would require removal of the associated dual-channel stick)
    • If we were to do that, we might need to adjust Varnish configuration to accomodate the change in available RAM

I'll check with the folks in both eqiad and codfw to see if they have any spare '32GB RDIMM, 3200MT/s, dual Rank 8Gb BASE x4' from decoms.

Updates:

Valerie shipped (3) replacement dimms, (1) for immediate use and (2) spares.

https://www.dhl.com/us-en/home/tracking.html?tracking-id=15%204938%206650&submit=1

Delivery ticket DEL0259735 opened. Once it lands I'll open a remote hands to have the ram swapped.