Page MenuHomePhabricator
Create Task
Maniphest T305423

cp5002 memory errors on DIMM A4
Closed, ResolvedPublic
Actions

Description

cp5002.eqsin.wmnet reports the following error from racadm getsel:

Record:      36
Date/Time:   04/05/2022 01:10:38
Source:      system
Severity:    Ok
Description: A problem was detected in Memory Reference Code (MRC).
-------------------------------------------------------------------------------
Record:      37
Date/Time:   04/05/2022 01:10:38
Source:      system
Severity:    Critical
Description: Multi-bit memory errors detected on a memory device at location(s) DIMM_A4.
-------------------------------------------------------------------------------

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedssinghT305423 cp5002 memory errors on DIMM A4
Unknown Object (Task)

Event Timeline

ssingh created this task.Apr 5 2022, 1:57 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 5 2022, 1:57 AM
Stashbot added a comment.Apr 5 2022, 1:59 AM

Mentioned in SAL (#wikimedia-operations) [2022-04-05T01:59:26Z] <sukhe@cumin2002> START - Cookbook sre.hosts.downtime for 30 days, 0:00:00 on cp5002.eqsin.wmnet with reason: downtimed because of hardware failure: T305423

Stashbot added a comment.Apr 5 2022, 1:59 AM

Mentioned in SAL (#wikimedia-operations) [2022-04-05T01:59:28Z] <sukhe@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 30 days, 0:00:00 on cp5002.eqsin.wmnet with reason: downtimed because of hardware failure: T305423

Maintenance_bot added a project: SRE.Apr 5 2022, 2:29 AM
Vgutierrez added a project: ops-eqsin.Apr 5 2022, 10:00 AM
KOfori subscribed.Apr 5 2022, 12:13 PM
wiki_willy assigned this task to RobH.Apr 8 2022, 5:44 PM
wiki_willy subscribed.

Hi @ssingh - since this server is out of warranty and due to be refreshed in a few quarters, do you still want us to purchase a replacement DIMM to keep it up and running in the meantime or are you able to wait it out? Thanks, Willy

RobH triaged this task as Medium priority.Apr 11 2022, 2:19 PM
RobH moved this task from Backlog to Hardware Failure / Repair on the ops-eqsin board.
ssingh added a comment.Apr 12 2022, 2:52 PM
In T305423#7841765, @wiki_willy wrote:

Hi @ssingh - since this server is out of warranty and due to be refreshed in a few quarters, do you still want us to purchase a replacement DIMM to keep it up and running in the meantime or are you able to wait it out? Thanks, Willy

Hi @wiki_willy: Sorry for the delayed response; I wanted to discuss this with the rest of the team. We decided that it makes sense to proceed with the replacement DIMM for now to keep the server running.

Thank you for checking!

wiki_willy added a comment.Apr 12 2022, 7:49 PM

Thanks @ssingh. Rob's working on sourcing the replacement DIMM, so we should have that sorted out soon, and will keep you in the loop via an adjacent procurement task. Thanks, Willy

RobH added a subtask: Unknown Object (Task).Apr 12 2022, 7:51 PM
RobH added a comment.May 2 2022, 7:34 PM

Jin will be onsite on May 4th @ 9AM Singapore Time to swap this memory out

Order Number - 1-216864938761

RobH reassigned this task from RobH to ssingh.May 4 2022, 12:06 AM
RobH subscribed.

This host has had its ram replaced and booted into the OS successfully, detecting all memory without errors.

When we were replacing the memory, it forgot its bios time/date as the CR battery on the mainboard has discharged. This is only an issue if power is entirely lost, requiring the date/time to be set again. I'm not sure that its worth the downtime and such to swap out mainboard batteries when these are due for replacement in Q3 of next fiscal.

@ssingh: Would you be the one to return this host to service? If so, can you do so and resolve this task when done? Thank you!

ssingh closed this task as Resolved.May 4 2022, 12:47 AM
In T305423#7901638, @RobH wrote:

This host has had its ram replaced and booted into the OS successfully, detecting all memory without errors.

When we were replacing the memory, it forgot its bios time/date as the CR battery on the mainboard has discharged. This is only an issue if power is entirely lost, requiring the date/time to be set again. I'm not sure that its worth the downtime and such to swap out mainboard batteries when these are due for replacement in Q3 of next fiscal.

@ssingh: Would you be the one to return this host to service? If so, can you do so and resolve this task when done? Thank you!

Thanks very much for your help @RobH! I have returned the host to service and we can consider this resolved. Much appreciated!

Papaul subscribed.May 4 2022, 12:53 AM

@ssingh the host is still has failed as status in netbox
https://netbox.wikimedia.org/dcim/devices/1611/

ssingh added a subscriber: PPaul.May 4 2022, 1:05 AM
In T305423#7901674, @Papaul wrote:

@ssingh the host is still has failed as status in netbox
https://netbox.wikimedia.org/dcim/devices/1611/

Thanks for letting me know @PPaul; resolving some errors on the host and then I will change the status in Netbox.

PPaul unsubscribed.May 5 2022, 9:50 PM
RobH closed subtask Unknown Object (Task) as Resolved.May 18 2022, 10:36 PM
This comment was removed by RobH.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL