Degraded RAID on cloudcephmon1004
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	ops-monitoring-bot
	Apr 22 2025, 12:21 PM

Project Tags

Referenced Files

None

Subscribers

Description

TASK AUTO-GENERATED by Nagios/Icinga RAID event handler

A degraded RAID (md) was detected on host cloudcephmon1004. An automatic snapshot of the current RAID status is attached below.

Please sync with the service owner to find the appropriate time window before actually replacing any failed hardware.

CRITICAL: State: degraded, Active: 3, Working: 3, Failed: 1, Spare: 0

$ sudo /usr/local/lib/nagios/plugins/get-raid-status-md
Personalities : [raid10] [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] 
md0 : active raid10 sda2[0] sdd2[3] sdb2[1](F) sdc2[2]
      1874534400 blocks super 1.2 512K chunks 2 near-copies [4/3] [U_UU]
      bitmap: 1/14 pages [4KB], 65536KB chunk

unused devices: <none>

Related Objects
Search...

Status	Subtype	Assigned	Task
Resolved		• aborrero	T392423 KernelErrors Server cloudcephmon1004 logged kernel errors
Resolved		VRiley-WMF	T392424 Degraded RAID on cloudcephmon1004
Resolved	Request	Jclark-ctr	T392458 hw troubleshooting: disk failure (sdb) on cloudcephmon1004

Event Timeline

ops-monitoring-bot created this task.Apr 22 2025, 12:21 PM

Restricted Application added a project: DC-Ops. · View Herald TranscriptApr 22 2025, 12:21 PM

taavi added a parent task: T392423: KernelErrors Server cloudcephmon1004 logged kernel errors.Apr 23 2025, 6:36 AM

First error from dmesg:

[Tue Apr 22 12:17:29 2025] sd 0:0:1:0: attempting task abort!scmd(0x000000008c6219ca), outstanding for 61144 ms & timeout 60000 ms
[Tue Apr 22 12:17:29 2025] sd 0:0:1:0: [sdb] tag#4322 CDB: Synchronize Cache(10) 35 00 00 00 00 00 00 00 00 00
[Tue Apr 22 12:17:29 2025] scsi target0:0:1: handle(0x0017), sas_address(0x3f4ee0806dabf30c), phy(12)
[Tue Apr 22 12:17:29 2025] scsi target0:0:1: enclosure logical id(0x3f4ee0806d092108), slot(3) 
[Tue Apr 22 12:17:29 2025] scsi target0:0:1: enclosure level(0x0001), connector name( C0  )
[Tue Apr 22 12:17:29 2025] sd 0:0:1:0: task abort: SUCCESS scmd(0x000000008c6219ca)
[Tue Apr 22 12:17:33 2025] mpt3sas_cm0: log_info(0x31111000): originator(PL), code(0x11), sub_code(0x1000)
[Tue Apr 22 12:17:37 2025] mpt3sas_cm0: log_info(0x31111000): originator(PL), code(0x11), sub_code(0x1000)
[Tue Apr 22 12:17:41 2025] mpt3sas_cm0: log_info(0x31111000): originator(PL), code(0x11), sub_code(0x1000)
[Tue Apr 22 12:17:44 2025] mpt3sas_cm0: log_info(0x31111000): originator(PL), code(0x11), sub_code(0x1000)
[Tue Apr 22 12:17:48 2025] mpt3sas_cm0: log_info(0x31111000): originator(PL), code(0x11), sub_code(0x1000)
[Tue Apr 22 12:17:48 2025] sd 0:0:1:0: [sdb] tag#6134 FAILED Result: hostbyte=DID_SOFT_ERROR driverbyte=DRIVER_OK cmd_age=79s
[Tue Apr 22 12:17:48 2025] sd 0:0:1:0: [sdb] tag#6134 CDB: Synchronize Cache(10) 35 00 00 00 00 00 00 00 00 00
[Tue Apr 22 12:17:48 2025] blk_update_request: I/O error, dev sdb, sector 585744 op 0x1:(WRITE) flags 0x20800 phys_seg 1 prio class 0
[Tue Apr 22 12:17:48 2025] md: super_written gets error=-5
[Tue Apr 22 12:17:48 2025] md/raid10:md0: Disk failure on sdb2, disabling device.
                           md/raid10:md0: Operation continuing on 3 devices.
[Tue Apr 22 12:21:07 2025] mpt3sas_cm0: log_info(0x31110e03): originator(PL), code(0x11), sub_code(0x0e03)
[Tue Apr 22 12:21:11 2025] mpt3sas_cm0: log_info(0x31111000): originator(PL), code(0x11), sub_code(0x1000)
[Tue Apr 22 12:21:12 2025] mpt3sas_cm0: log_info(0x31110635): originator(PL), code(0x11), sub_code(0x0635)
[Tue Apr 22 12:21:16 2025] mpt3sas_cm0: log_info(0x31110e03): originator(PL), code(0x11), sub_code(0x0e03)
[Tue Apr 22 12:21:19 2025] mpt3sas_cm0: log_info(0x31111000): originator(PL), code(0x11), sub_code(0x1000)
[Tue Apr 22 12:21:23 2025] mpt3sas_cm0: log_info(0x31111000): originator(PL), code(0x11), sub_code(0x1000)
[Tue Apr 22 12:21:23 2025] sd 0:0:1:0: [sdb] tag#1303 FAILED Result: hostbyte=DID_SOFT_ERROR driverbyte=DRIVER_OK cmd_age=19s
[Tue Apr 22 12:21:23 2025] sd 0:0:1:0: [sdb] tag#1303 CDB: Read(10) 28 00 00 00 00 00 00 01 00 00
[Tue Apr 22 12:21:23 2025] blk_update_request: I/O error, dev sdb, sector 0 op 0x0:(READ) flags 0x0 phys_seg 32 prio class 0
...

It keeps showing some of those errors every few minutes:

...
[Wed Apr 23 03:22:40 2025] blk_update_request: I/O error, dev sdb, sector 585728 op 0x0:(READ) flags 0x0 phys_seg 32 prio class 0
[Wed Apr 23 03:22:40 2025] sd 0:0:1:0: [sdb] tag#5537 FAILED Result: hostbyte=DID_SOFT_ERROR driverbyte=DRIVER_OK cmd_age=19s   
[Wed Apr 23 03:22:40 2025] sd 0:0:1:0: [sdb] tag#5537 CDB: Read(10) 28 00 00 08 ef 80 00 00 08 00         
[Wed Apr 23 03:22:40 2025] blk_update_request: I/O error, dev sdb, sector 585600 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[Wed Apr 23 03:22:40 2025] Buffer I/O error on dev sdb1, logical block 72944, async page read                                
[Wed Apr 23 03:38:41 2025] mpt3sas_cm0: log_info(0x31110e03): originator(PL), code(0x11), sub_code(0x0e03)                             
[Wed Apr 23 03:38:41 2025] program smartctl is using a deprecated SCSI ioctl, please convert it to SG_IO                       
[Wed Apr 23 03:38:45 2025] mpt3sas_cm0: log_info(0x31111000): originator(PL), code(0x11), sub_code(0x1000)                             
[Wed Apr 23 03:38:49 2025] mpt3sas_cm0: log_info(0x31111000): originator(PL), code(0x11), sub_code(0x1000)                          
[Wed Apr 23 03:38:52 2025] mpt3sas_cm0: log_info(0x31111000): originator(PL), code(0x11), sub_code(0x1000)                      
[Wed Apr 23 03:38:56 2025] mpt3sas_cm0: log_info(0x31111000): originator(PL), code(0x11), sub_code(0
...

dcaro mentioned this in T392458: hw troubleshooting: disk failure (sdb) on cloudcephmon1004.Apr 23 2025, 8:33 AM

dcaro added a subtask: T392458: hw troubleshooting: disk failure (sdb) on cloudcephmon1004.

root@cloudcephmon1004:~# sudo mdadm --detail /dev/md0
/dev/md0:
           Version : 1.2
     Creation Time : Tue Nov 26 11:34:32 2024
        Raid Level : raid10
        Array Size : 1874534400 (1787.70 GiB 1919.52 GB)
     Used Dev Size : 937267200 (893.85 GiB 959.76 GB)
      Raid Devices : 4
     Total Devices : 4
       Persistence : Superblock is persistent

     Intent Bitmap : Internal

       Update Time : Wed Apr 23 08:35:05 2025
             State : active, degraded 
    Active Devices : 3
   Working Devices : 3
    Failed Devices : 1
     Spare Devices : 0

            Layout : near=2
        Chunk Size : 512K

Consistency Policy : bitmap

              Name : cloudcephmon1004:0  (local to host cloudcephmon1004)
              UUID : 97219653:c33e3546:d9a2b57f:a2125219
            Events : 351460

    Number   Major   Minor   RaidDevice State
       0       8        2        0      active sync set-A   /dev/sda2
       -       0        0        1      removed
       2       8       34        2      active sync set-A   /dev/sdc2
       3       8       50        3      active sync set-B   /dev/sdd2

       1       8       18        -      faulty   /dev/sdb2

Papaul moved this task from Backlog to Hardware Failure / Troubleshoot on the ops-eqiad board.Apr 28 2025, 3:45 AM

wiki_willy added subscribers: VRiley-WMF, Jclark-ctr.Apr 28 2025, 6:00 PM

Created a Dell service request for this Service Request 209252181.

Updated. Closing ticket, as this is a duplicate of T392458

VRiley-WMF closed this task as Resolved.Apr 30 2025, 6:42 AM

VRiley-WMF claimed this task.

Jclark-ctr closed subtask T392458: hw troubleshooting: disk failure (sdb) on cloudcephmon1004 as Resolved.May 1 2025, 9:37 PM

andrea.denisse reopened subtask T392458: hw troubleshooting: disk failure (sdb) on cloudcephmon1004 as Open.May 26 2025, 3:26 PM

andrea.denisse closed subtask T392458: hw troubleshooting: disk failure (sdb) on cloudcephmon1004 as Resolved.May 26 2025, 6:55 PM