hw troubleshooting: disk replacement for an-worker1110.eqiad.wmnet
Closed, ResolvedPublicRequest
Actions

Assigned To

Authored By

	BTullis
	May 18 2023, 11:05 AM

Description

- Provide FQDN of system.
- If other than a hard drive issue, please depool the machine (and confirm that it’s been depooled) for us to work on it. If not, please provide time frame for us to take the machine down.
- Put system into a failed state in Netbox. n/a as the machine is working with the remaining 11 disks
- Provide urgency of request, along with justification (redundancy, dependencies, etc)
- Describe issue and/or attach hardware failure log. (Refer to https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook if you need help)
- Assign correct project tag and appropriate owner (based on above). Also, please ensure the service owners of the host(s) are added as subscribers to provide any additional input.

Please would you replace the 4TB hard drive in an-worker1110 slot 4? It can be hot-swapped at any time.

The service owner Data-Engineering ticket is T336929: Disk failure on an-worker1110 and that's where I have detailed the investigation so far.

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Resolved		BTullis	T336929 Disk failure on an-worker1110
		Resolved	Request	Jclark-ctr	T336930 hw troubleshooting: disk replacement for an-worker1110.eqiad.wmnet

Event Timeline

BTullis created this task.May 18 2023, 11:05 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 18 2023, 11:05 AM

BTullis added a parent task: T336929: Disk failure on an-worker1110.May 18 2023, 11:07 AM

I have marked the drive as offline and set the slot to blink.

btullis@an-worker1110:~$ sudo megacli -PDOffline -PhysDrv '[32:4]' -a0
                                     
Adapter: 0: EnclId-32 SlotId-4 state changed to OffLine.

Exit Code: 0x00
btullis@an-worker1110:~$ sudo megacli -PDLocate -PhysDrv '[32:4]' -a0
                                     
Adapter: 0: Device at EnclId-32 SlotId-4  -- PD Locate Start Command was successfully sent to Firmware 

Exit Code: 0x00

Hopefully this will help to identify the failed drive.

BTullis mentioned this in T336929: Disk failure on an-worker1110.May 18 2023, 11:26 AM

@Jclark-ctr - Do you know when you might be able to look at this issue? It's not that it's urgent, just looking to keep on top of the tickets. Thanks.