Page MenuHomePhabricator

Degraded RAID on an-worker1199
Closed, ResolvedPublic

Description

TASK AUTO-GENERATED by Nagios/Icinga RAID event handler

A degraded RAID (broadcom) was detected on host an-worker1199. An automatic snapshot of the current RAID status is attached below.

Please sync with the service owner to find the appropriate time window before actually replacing any failed hardware.

communication: 0 OK : controller: 1 Needs Attention : physical_disk: 1 Failed : virtual_disk: 1 OfLn : bbu: 0 OK : enclosure: 0 OK : CLI Version = 007.1910.0000.0000 Oct 08, 2021

$ sudo /usr/local/lib/nagios/plugins/get-raid-status-broadcom
Failed to execute '['/usr/lib/nagios/plugins/check_nrpe', '-4', '-H', 'an-worker1199', '-c', 'get_raid_status_broadcom']': RETCODE: 2
STDOUT:
communication: 0 OK ; controller: 1 Needs Attention ; physical_disk: 1 Failed ; virtual_disk: 1 OfLn ; bbu: 0 OK ; enclosure: 0 OK ; CLI Version = 007.1910.0000.0000 Oct 08, 2021
Operating system = Linux 5.10.0-34-amd64
Controller = 0
Status = Success
Description = Show Drive Group Succeeded


TOPOLOGY :
========

-----------------------------------------------------------------------------
DG Arr Row EID:Slot DID Type  State BT       Size PDC  PI SED DS3  FSpace TR 
-----------------------------------------------------------------------------
 0 -   -   -        -   RAID1 Optl  N  446.625 GB enbl N  N   dflt N      N  
 0 0   -   -        -   RAID1 Optl  N  446.625 GB enbl N  N   dflt N      N  
 0 0   0   251:0    5   DRIVE Onln  N  446.625 GB enbl N  N   dflt -      N  
 0 0   1   251:1    7   DRIVE Onln  N  446.625 GB enbl N  N   dflt -      N  
 1 -   -   -        -   RAID0 Optl  N    7.276 TB enbl N  N   dflt N      N  
 1 0   -   -        -   RAID0 Optl  N    7.276 TB enbl N  N   dflt N      N  
 1 0   0   252:0    0   DRIVE Onln  N    7.276 TB enbl N  N   dflt -      N  
 2 -   -   -        -   RAID0 Optl  N    7.276 TB enbl N  N   dflt N      N  
 2 0   -   -        -   RAID0 Optl  N    7.276 TB enbl N  N   dflt N      N  
 2 0   0   252:1    6   DRIVE Onln  N    7.276 TB enbl N  N   dflt -      N  
 3 -   -   -        -   RAID0 OfLn  N    7.276 TB enbl N  N   dflt N      N  
 3 0   -   -        -   RAID0 Dgrd  N    7.276 TB enbl N  N   dflt N      N  
 4 -   -   -        -   RAID0 Optl  N    7.276 TB enbl N  N   dflt N      N  
 4 0   -   -        -   RAID0 Optl  N    7.276 TB enbl N  N   dflt N      N  
 4 0   0   252:3    9   DRIVE Onln  N    7.276 TB enbl N  N   dflt -      N  
 5 -   -   -        -   RAID0 Optl  N    7.276 TB enbl N  N   dflt N      N  
 5 0   -   -        -   RAID0 Optl  N    7.276 TB enbl N  N   dflt N      N  
 5 0   0   252:4    11  DRIVE Onln  N    7.276 TB enbl N  N   dflt -      N  
 6 -   -   -        -   RAID0 Optl  N    7.276 TB enbl N  N   dflt N      N  
 6 0   -   -        -   RAID0 Optl  N    7.276 TB enbl N  N   dflt N      N  
 6 0   0   252:5    10  DRIVE Onln  N    7.276 TB enbl N  N   dflt -      N  
 7 -   -   -        -   RAID0 Optl  N    7.276 TB enbl N  N   dflt N      N  
 7 0   -   -        -   RAID0 Optl  N    7.276 TB enbl N  N   dflt N      N  
 7 0   0   252:6    12  DRIVE Onln  N    7.276 TB enbl N  N   dflt -      N  
 8 -   -   -        -   RAID0 Optl  N    7.276 TB enbl N  N   dflt N      N  
 8 0   -   -        -   RAID0 Optl  N    7.276 TB enbl N  N   dflt N      N  
 8 0   0   252:7    13  DRIVE Onln  N    7.276 TB enbl N  N   dflt -      N  
 9 -   -   -        -   RAID0 Optl  N    7.276 TB enbl N  N   dflt N      N  
 9 0   -   -        -   RAID0 Optl  N    7.276 TB enbl N  N   dflt N      N  
 9 0   0   252:8    1   DRIVE Onln  N    7.276 TB enbl N  N   dflt -      N  
10 -   -   -        -   RAID0 Optl  N    7.276 TB enbl N  N   dflt N      N  
10 0   -   -        -   RAID0 Optl  N    7.276 TB enbl N  N   dflt N      N  
10 0   0   252:9    3   DRIVE Onln  N    7.276 TB enbl N  N   dflt -      N  
11 -   -   -        -   RAID0 Optl  N    7.276 TB enbl N  N   dflt N      N  
11 0   -   -        -   RAID0 Optl  N    7.276 TB enbl N  N   dflt N      N  
11 0   0   252:10   2   DRIVE Onln  N    7.276 TB enbl N  N   dflt -      N  
12 -   -   -        -   RAID0 Optl  N    7.276 TB enbl N  N   dflt N      N  
12 0   -   -        -   RAID0 Optl  N    7.276 TB enbl N  N   dflt N      N  
12 0   0   252:11   4   DRIVE Onln  N    7.276 TB enbl N  N   dflt -      N  
-----------------------------------------------------------------------------

DG=Disk Group Index|Arr=Array Index|Row=Row Index|EID=Enclosure Device ID
DID=Device ID|Type=Drive Type|Onln=Online|Rbld=Rebuild|Optl=Optimal|Dgrd=Degraded
Pdgd=Partially degraded|Offln=Offline|BT=Background Task Active
PDC=PD Cache|PI=Protection Info|SED=Self Encrypting Drive|Frgn=Foreign
DS3=Dimmer Switch 3|dflt=Default|Msng=Missing|FSpace=Free Space Present
TR=Transport Ready






STDERR:
None

Event Timeline

Created a Service Request ticket with Dell - SR218125316

Opened inbound ticket 1-253742292236

Make sure to upload tsr reports when submitting tickets

Work Order: SR218125316
Denial Notes
Thank you for submitting the request. We would require more details in order to proceed with your request. Please provide Error Logs (Support Assist/TSR Logs) from the diagnostics run to determine the failure with the part.

Resubmitted SR with TSR attached.

Checked on this ticket, the order is processing

@BTullis we have recieved the drive for this unit. Is there a time for us to replace this?

Yes, please. Feel free to go ahead. Apologies for the delay.

VRiley-WMF changed the task status from Open to In Progress.Nov 18 2025, 5:53 PM

Swapping now

Disk has been swapped

@VRiley-WMF, you should leave this ticket open until @BTullis has had the opportunity to complete the final step of adding the disk to the RAID.

Thanks. I've unmounted /dev/sdl1 which was still showing errors on dmesg -T so you can feel free to swap the drive now.
Here is the corrent state of the physical disks.

--------------------------------------------------------------------------------------
EID:Slt DID State  DG       Size Intf Med SED PI SeSz Model                   Sp Type 
--------------------------------------------------------------------------------------
251:0     5 Onln    0 446.625 GB SATA SSD N   N  512B MTFDDAK480TGA-1BC1ZABDA U  -    
251:1     7 Onln    0 446.625 GB SATA SSD N   N  512B MTFDDAK480TGA-1BC1ZABDA U  -    
252:0     0 Onln    1   7.276 TB SATA HDD N   N  512B TOSHIBA MG08ADA800EY    U  -    
252:1     6 Onln    2   7.276 TB SATA HDD N   N  512B TOSHIBA MG08ADA800EY    U  -    
252:2    14 UGood   -   7.276 TB SATA HDD N   N  512B ST8000NM023B-2TJ133     U  -    
252:3     9 Onln    3   7.276 TB SATA HDD N   N  512B TOSHIBA MG08ADA800EY    U  -    
252:4    11 Onln    4   7.276 TB SATA HDD N   N  512B TOSHIBA MG08ADA800EY    U  -    
252:5    10 Onln    5   7.276 TB SATA HDD N   N  512B TOSHIBA MG08ADA800EY    U  -    
252:6    12 Onln    6   7.276 TB SATA HDD N   N  512B TOSHIBA MG08ADA800EY    U  -    
252:7    13 Onln    7   7.276 TB SATA HDD N   N  512B TOSHIBA MG08ADA800EY    U  -    
252:8     1 Onln    8   7.276 TB SATA HDD N   N  512B TOSHIBA MG08ADA800EY    U  -    
252:9     3 Onln    9   7.276 TB SATA HDD N   N  512B TOSHIBA MG08ADA800EY    U  -    
252:10    2 Failed 10   7.276 TB SATA HDD N   N  512B TOSHIBA MG08ADA8        U  -    
252:11    4 Onln   11   7.276 TB SATA HDD N   N  512B TOSHIBA MG08ADA800EY    U  -    
--------------------------------------------------------------------------------------

One Failed and one UGood.