Page MenuHomePhabricator

hw troubleshooting: disk in slot 10 for an-worker1194
Closed, ResolvedPublicRequest

Description

  • - Provide FQDN of system.
  • - If other than a hard drive issue, please depool the machine (and confirm that it’s been depooled) for us to work on it. If not, please provide time frame for us to take the machine down.
  • - Put system into a failed state in Netbox. n/a
  • - Provide urgency of request, along with justification (redundancy, dependencies, etc) - Not urgent.
  • - Describe issue and/or attach hardware failure log. (Refer to https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook if you need help)
  • - Assign correct project tag and appropriate owner (based on above). Also, please ensure the service owners of the host(s) are added as subscribers to provide any additional input.

The disk in slot 10 of a new server, an-worker1194 seems not to have been detected. We can verify this from sudo perccli64 /c0 show all on the host.

image.png (836×423 px, 81 KB)

Please could you have a look at it and re-seat or replace it as necessary? Thanks.

Event Timeline

Replaced failed drive

RKemper subscribed.

Slot 252:10 failed again D:

During T411568 rolling reboots, an-worker1194 failed to reboot (systemd hung waiting for /dev/disk/by-label/hadoop-e). The drive in slot 252:10 was showing up as UGood/Foreign (no VD configured). I cleared the foreign config, created a new RAID0 VD, partitioned, and ran mkfs ext4. On mount, the drive threw I/O errors (rejecting I/O to dead device) and vanished from the controller entirely (perccli64 /c0 /e252 /s10 show -> 'Drive not found').

For now, I removed hadoop-e from fstab to allow the host to boot with 11 drives.

We should figure out if this is a backplane issue or a faulty replacement drive. Or possibly even just leave this host with 11 drives in perpetuity; however this host is only about a year old so we should probably try to restore it fully.

I don't have the best understanding of this stuff, but I think if it got through the mkfs (sequential write) but failed the mount (random read) then the backplane explanation is probably more likely?

Thoughts @BTullis?

I’m do have a spare drive to swap it with let me know when it is Removed so I can swap @RKemper

This has been done as part of the investigation into T415002