codfw: db2079 memory issue on DIMM B8
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Papaul
	May 26 2021, 4:48 PM

Description

db2079 has a bad DIMM (B8)

 	Thu Apr 22 2021 10:45:58	Correctable memory error rate exceeded for DIMM_B8.	
	Thu Apr 22 2021 10:22:56	Correctable memory error rate exceeded for DIMM_B8.

I will like to swap B8 with A8 for testing before replacing.

Thanks

Related Objects

Mentioned Here: T216240: Reboot, upgrade firmware and kernel of db1096-db1106, db2071-db2092

Event Timeline

Papaul created this task.May 26 2021, 4:48 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 26 2021, 4:48 PM

Papaul triaged this task as Medium priority.May 26 2021, 4:48 PM

This is s8 master, so it needs some coordination. Let me know a day/time when you'd like to tackle this and I can have the host ready for you!

Marostegui moved this task from Triage to Refine on the DBA board.May 26 2021, 4:50 PM

Maintenance_bot added a project: SRE.May 26 2021, 5:45 PM

Papaul moved this task from Backlog to Hardware Failure / Troubleshoot on the ops-codfw board.May 31 2021, 11:36 PM

@Marostegui hello you can go ahead and depool the server i will be on site in about an hour.

Thanks

Excellent, thanks @Papaul

Mentioned in SAL (#wikimedia-operations) [2021-06-01T13:56:32Z] <marostegui> Stop mysql on db2079 (codfw master) - T283743

db2079 is off and ready for you @Papaul

Swapped DIMM B8 with DIMM A8 we will see if we do see the issue on DIMM A8 . If we do, I will use one of the DIMM from one if the Decom servers .

Resolving this task for now

On boot, we are hitting T216240, @Papaul let's get firmware and bios upgraded please

Firmware upgrade complete

MySQL started - thanks Papaul!

codfw: db2079 memory issue on DIMM B8Closed, ResolvedPublicActions

Description

Related Objects

Event Timeline

codfw: db2079 memory issue on DIMM B8
Closed, ResolvedPublic
Actions