Degraded RAID on druid1011
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	ops-monitoring-bot
	Oct 3 2025, 9:54 PM

Project Tags

Referenced Files

None

Subscribers

Description

TASK AUTO-GENERATED by Nagios/Icinga RAID event handler

A degraded RAID (md) was detected on host druid1011. An automatic snapshot of the current RAID status is attached below.

Please sync with the service owner to find the appropriate time window before actually replacing any failed hardware.

CRITICAL: State: degraded, Active: 7, Working: 7, Failed: 1, Spare: 0

$ sudo /usr/local/lib/nagios/plugins/get-raid-status-md
Personalities : [raid10] [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] 
md0 : active raid10 sde2[4](F) sdc2[2] sdd2[3] sdg2[7] sda2[0] sdb2[1] sdf2[5] sdh2[6]
      3749068800 blocks super 1.2 512K chunks 2 near-copies [8/7] [UUUU_UUU]
      bitmap: 1/28 pages [4KB], 65536KB chunk

unused devices: <none>

Event Timeline

ops-monitoring-bot created this task.Oct 3 2025, 9:54 PM

Restricted Application added a project: DC-Ops. · View Herald TranscriptOct 3 2025, 9:54 PM

Jclark-ctr moved this task from Backlog to Hardware Failure / Troubleshoot on the ops-eqiad board.Oct 5 2025, 11:41 PM

This server is currently out of warranty. It appears to be a non-RAID server. Could we pull a drive from a recently decommissioned server?

Jclark-ctr claimed this task.Oct 6 2025, 11:33 AM

• Stevemunene subscribed.Oct 6 2025, 12:24 PM

Gehel added a project: Essential-Work.Oct 7 2025, 8:51 AM

@BTullis can you assist with this

OK, this is a software RAID10 volume. It looks like it is drive /dev/sde that has failed.

btullis@druid1011:~$ cat /proc/mdstat 
Personalities : [raid10] [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] 
md0 : active raid10 sde2[4](F) sdc2[2] sdd2[3] sdg2[7] sda2[0] sdb2[1] sdf2[5] sdh2[6]
      3749068800 blocks super 1.2 512K chunks 2 near-copies [8/7] [UUUU_UUU]
      bitmap: 20/28 pages [80KB], 65536KB chunk

unused devices: <none>
btullis@druid1011:~$

It should be safe to pull the drive and hot-swap a new one in, but I will probably have to add the new drive to the array manually.

@Jclark-ctr are you able to identify which physical drive is faulty here? I guess it's the one that isn't blinking the same as all the rest, but there is no way to identify the slot as there would be on a hardware RAID server.

I will work on this tomorrow since i am using an older drive i want to make sure it is wiped prior to installing.

@BTullis Replaced Failed drive and converted to non-raid state via idrac.

while logged in updated idrac from 6.00.30.00 to 7.00.00.182

Thanks @Jclark-ctr - Looks good. I can see that the drive showed up as /dev/sde and had no partition table.

[Fri Oct 10 13:14:56 2025] scsi 0:2:4:0: Direct-Access     ATA      SSDSC2KB960G8R   DL63 PQ: 0 ANSI: 6
[Fri Oct 10 13:14:56 2025] sd 0:2:4:0: Attached scsi generic sg4 type 0
[Fri Oct 10 13:14:56 2025] sd 0:2:4:0: [sde] 1875385008 512-byte logical blocks: (960 GB/894 GiB)
[Fri Oct 10 13:14:56 2025] sd 0:2:4:0: [sde] 4096-byte physical blocks
[Fri Oct 10 13:14:56 2025] sd 0:2:4:0: [sde] Write Protect is off
[Fri Oct 10 13:14:56 2025] sd 0:2:4:0: [sde] Mode Sense: 6b 00 10 08
[Fri Oct 10 13:14:56 2025] sd 0:2:4:0: [sde] Write cache: enabled, read cache: enabled, supports DPO and FUA
[Fri Oct 10 13:14:56 2025] sd 0:2:4:0: [sde] Attached SCSI disk

I checked the other devices.

btullis@druid1011:~$ lsblk
NAME           MAJ:MIN RM   SIZE RO TYPE   MOUNTPOINT
sda              8:0    0 894.3G  0 disk   
├─sda1           8:1    0   285M  0 part   
└─sda2           8:2    0   894G  0 part   
  └─md0          9:0    0   3.5T  0 raid10 
    ├─vg0-swap 253:0    0   976M  0 lvm    [SWAP]
    ├─vg0-root 253:1    0  74.5G  0 lvm    /
    └─vg0-srv  253:2    0   2.7T  0 lvm    /srv
sdb              8:16   0 894.3G  0 disk   
├─sdb1           8:17   0   285M  0 part   
└─sdb2           8:18   0   894G  0 part   
  └─md0          9:0    0   3.5T  0 raid10 
    ├─vg0-swap 253:0    0   976M  0 lvm    [SWAP]
    ├─vg0-root 253:1    0  74.5G  0 lvm    /
    └─vg0-srv  253:2    0   2.7T  0 lvm    /srv
sdc              8:32   0 894.3G  0 disk   
├─sdc1           8:33   0   285M  0 part   
└─sdc2           8:34   0   894G  0 part   
  └─md0          9:0    0   3.5T  0 raid10 
    ├─vg0-swap 253:0    0   976M  0 lvm    [SWAP]
    ├─vg0-root 253:1    0  74.5G  0 lvm    /
    └─vg0-srv  253:2    0   2.7T  0 lvm    /srv
sdd              8:48   0 894.3G  0 disk   
├─sdd1           8:49   0   285M  0 part   
└─sdd2           8:50   0   894G  0 part   
  └─md0          9:0    0   3.5T  0 raid10 
    ├─vg0-swap 253:0    0   976M  0 lvm    [SWAP]
    ├─vg0-root 253:1    0  74.5G  0 lvm    /
    └─vg0-srv  253:2    0   2.7T  0 lvm    /srv
sde              8:64   0 894.3G  0 disk   
sdf              8:80   0 894.3G  0 disk   
├─sdf1           8:81   0   285M  0 part   
└─sdf2           8:82   0   894G  0 part   
  └─md0          9:0    0   3.5T  0 raid10 
    ├─vg0-swap 253:0    0   976M  0 lvm    [SWAP]
    ├─vg0-root 253:1    0  74.5G  0 lvm    /
    └─vg0-srv  253:2    0   2.7T  0 lvm    /srv
sdg              8:96   0 894.3G  0 disk   
├─sdg1           8:97   0   285M  0 part   
└─sdg2           8:98   0   894G  0 part   
  └─md0          9:0    0   3.5T  0 raid10 
    ├─vg0-swap 253:0    0   976M  0 lvm    [SWAP]
    ├─vg0-root 253:1    0  74.5G  0 lvm    /
    └─vg0-srv  253:2    0   2.7T  0 lvm    /srv
sdh              8:112  0 894.3G  0 disk   
├─sdh1           8:113  0   285M  0 part   
└─sdh2           8:114  0   894G  0 part   
  └─md0          9:0    0   3.5T  0 raid10 
    ├─vg0-swap 253:0    0   976M  0 lvm    [SWAP]
    ├─vg0-root 253:1    0  74.5G  0 lvm    /
    └─vg0-srv  253:2    0   2.7T  0 lvm    /srv

I took a copy of the partition table from /dev/sdh and applied it to /dev/sde

root@druid1011:~# sfdisk -d /dev/sdh | sfdisk /dev/sde
Checking that no-one is using this disk right now ... OK

Disk /dev/sde: 894.25 GiB, 960197124096 bytes, 1875385008 sectors
Disk model: SSDSC2KB960G8R  
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes

>>> Script header accepted.
>>> Script header accepted.
>>> Script header accepted.
>>> Script header accepted.
>>> Script header accepted.
>>> Script header accepted.
>>> Script header accepted.
>>> Created a new GPT disklabel (GUID: 999AD681-498B-45DB-BD80-444B9A1F1C70).
/dev/sde1: Created a new partition 1 of type 'BIOS boot' and of size 285 MiB.
/dev/sde2: Created a new partition 2 of type 'Linux RAID' and of size 894 GiB.
/dev/sde3: Done.

New situation:
Disklabel type: gpt
Disk identifier: 999AD681-498B-45DB-BD80-444B9A1F1C70

Device      Start        End    Sectors  Size Type
/dev/sde1    2048     585727     583680  285M BIOS boot
/dev/sde2  585728 1875384319 1874798592  894G Linux RAID

The partition table has been altered.
Calling ioctl() to re-read partition table.
Syncing disks.
root@druid1011:~#

Then I added /dev/sde2 to the /dev/md0 array.

root@druid1011:~# mdadm --manage /dev/md0 --add /dev/sde2
mdadm: added /dev/sde2
root@druid1011:~#

The array is now rebuilding.

root@druid1011:~# cat /proc/mdstat 
Personalities : [raid10] [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] 
md0 : active raid10 sde2[8] sdc2[2] sdd2[3] sdg2[7] sda2[0] sdb2[1] sdf2[5] sdh2[6]
      3749068800 blocks super 1.2 512K chunks 2 near-copies [8/7] [UUUU_UUU]
      [>....................]  recovery =  0.3% (3559168/937267200) finish=742.3min speed=20961K/sec
      bitmap: 20/28 pages [80KB], 65536KB chunk

unused devices: <none>
root@druid1011:~#

Maybe 12 hours or so to run.
I'll keep the ticket open over the weekend, then check again on Monday.

BTullis moved this task from Backlog - project to Blocked/Waiting on the Data-Platform-SRE (2025.09.26 - 2025.10.17) board.Oct 10 2025, 3:41 PM

This is now complete.

root@druid1011:~# cat /proc/mdstat 
Personalities : [raid10] [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] 
md0 : active raid10 sde2[8] sdc2[2] sdd2[3] sdg2[7] sda2[0] sdb2[1] sdf2[5] sdh2[6]
      3749068800 blocks super 1.2 512K chunks 2 near-copies [8/8] [UUUUUUUU]
      bitmap: 1/28 pages [4KB], 65536KB chunk

unused devices: <none>
root@druid1011:~#

Gehel moved this task from Done to Reported on the Data-Platform-SRE (2025.09.26 - 2025.10.17) board.Oct 17 2025, 8:30 AM

Jclark-ctr removed BTullis as the assignee of this task.Oct 22 2025, 4:55 PM

Jclark-ctr claimed this task.

Degraded RAID on druid1011Closed, ResolvedPublicActions

Description

Event Timeline

Degraded RAID on druid1011
Closed, ResolvedPublic
Actions