Broken disk on thanos-be1003
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	MoritzMuehlenhoff
	Aug 28 2023, 1:46 PM

Description

thanos-be1003 has a disk error flagging in Icinga:

DISK CRITICAL - /srv/swift-storage/sdn1 is not accessible: Input/output error

The host is slightly out of warranty, but maybe we have a spare disk?

Event Timeline

MoritzMuehlenhoff created this task.Aug 28 2023, 1:46 PM

MoritzMuehlenhoff triaged this task as Medium priority.

Papaul moved this task from Backlog to Hardware Failure / Troubleshoot on the ops-eqiad board.Aug 28 2023, 10:39 PM

Replaced hdd with spare hdd

2023-08-29 18:36:50 PDR3 Disk 11 in Backplane 1 of Integrated RAID Controller 1 is not functioning correctly. Part Number = TH0XPJ47SGT0003K01PDA00

imported foreign configuration to raid controller

Jclark-ctr closed this task as Resolved.Aug 29 2023, 6:50 PM

Mentioned in SAL (#wikimedia-operations) [2023-08-30T08:51:10Z] <Emperor> stopping puppet to fix broken drive labelling after disk swap thanos-be1003 T345079

puppet had spotted the new drive was /dev/sdm so made a new filesystem labelled swift-sdm1 on it (even though there was already an FS with that label mounted and in use). I fixed this up by hand, and the new drive is now serving as swift-sdn1.

MatthewVernon added a project: SRE-swift-storage.Aug 30 2023, 8:58 AM

Broken disk on thanos-be1003Closed, ResolvedPublicActions

Description

Event Timeline

Broken disk on thanos-be1003
Closed, ResolvedPublic
Actions