Page MenuHomePhabricator

hw troubleshooting: Memory errors for ml-serve2002.codfw.wmnet
Closed, ResolvedPublicRequest

Description

  • - Provide FQDN of system.
  • - If other than a hard drive issue, please depool the machine (and confirm that it’s been depooled) for us to work on it. If not, please provide time frame for us to take the machine down.

Machine is currently serving but can be taken out of service on very short notice.

  • - Provide urgency of request, along with justification (redundancy, dependencies, etc)

Not super urgent, since machine is still working with reduced capacity.

During reboot, BIOS will halt with error messages about failed DIMMs.

  • - Assign correct project tag and appropriate owner (based on above). Also, please ensure the service owners of the host(s) are added as subscribers to provide any additional input.

racadm getsel log:

The DIMMs affected are A1 and B2. Usually, cold booting the machine helps for a while. The two DIMMs have been doing this about once a year since 2021. I doubt they're truly broken, but may just need a re-seat (or it's the PCB the memory sockets are on, but the "fix" is the same).

The machine can be taken out of service/shutdown on short notice during CEST daytime. If that does not fit the DC work schedule, we can of course take it out for a few days.

Event Timeline

Mentioned in SAL (#wikimedia-operations) [2026-07-29T15:10:41Z] <klausman@cumin1003> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on ml-serve2002.codfw.wmnet with reason: T433476

Jhancock.wm claimed this task.
Jhancock.wm subscribed.

tobias downtimed the server for me and i reseated the DIMM with errors. Server threw a new error on boot.

2026-07-28 23:50:45 	UEFI0098 	The memory Built-In Self-test (BIST) has detected one or more errors on the DIMM installed on memory slot: B1. As a result, the corresponding DIMM has been disabled.	
2026-07-28 23:50:45 	UEFI0339 	The Dual Inline Memory Module (DIMM) in the memory slot B1 is disabled because of initialization errors caused by uncorrectable memory errors, invalid configuration, and others.

tried reseating it again. no results. upgraded the bios and idrac firmware. error persists. did some research on the codes. recommended fix is cleaning the dimm with isoprohyl alcohol, but don't have any on hand. pulled out the DIMM and set aside to clean later. Installed dimm from decommed servers. all errors resolved.