codfw: cp2038 Correctable memory error on DIMM A3
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Papaul
	May 16 2022, 5:10 PM

Description

There is a potential bad memory on cp2038. I will like for the system to be depool if possible for me to swap DINM A3 with DIMM B3
Thanks.

Correctable memory error logging disabled for a memory device at location DIMM_A3.

Details

Other Assignee: Vgutierrez

Event Timeline

Papaul created this task.May 16 2022, 5:10 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 16 2022, 5:10 PM

Papaul triaged this task as Medium priority.May 16 2022, 5:10 PM

Papaul moved this task from Backlog to Hardware Failure / Troubleshoot on the ops-codfw board.

Maintenance_bot added a project: SRE.May 16 2022, 5:29 PM

ssingh updated Other Assignee, added: ssingh.May 20 2022, 1:27 AM

Mentioned in SAL (#wikimedia-operations) [2022-05-20T13:24:53Z] <sukhe@cumin2002> START - Cookbook sre.hosts.downtime for 7 days, 0:00:00 on cp2038.codfw.wmnet with reason: downtimed because of DIMM replacement: T308459

Mentioned in SAL (#wikimedia-operations) [2022-05-20T13:24:58Z] <sukhe@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on cp2038.codfw.wmnet with reason: downtimed because of DIMM replacement: T308459

Hi @Papaul: Thanks for letting us know! The host is depooled and downtimed and so please proceed whenever you want. Thanks!

@ssingh thanks will work on it when back on site next week

Vgutierrez updated Other Assignee, added: Vgutierrez; removed: ssingh.May 23 2022, 12:32 PM

Vgutierrez subscribed.

I Swapped DIMMM A3 with DIMM B3 . No error showing on DIMMB3 for now. I upgrade also IDRAC from version 4.10 to 5.00. Resolving this task for now.
@Vgutierrez you can put the server back in service for now. Thanks

Mentioned in SAL (#wikimedia-operations) [2022-05-23T15:39:13Z] <vgutierrez> pool cp2038 - T308459

@Papaul thanks, cp2038 is happily serving traffic :)

codfw: cp2038 Correctable memory error on DIMM A3Closed, ResolvedPublicActions

Description

Details

Event Timeline

codfw: cp2038 Correctable memory error on DIMM A3
Closed, ResolvedPublic
Actions