Page MenuHomePhabricator

Degraded RAID on an-worker1199
Closed, ResolvedPublic

Description

TASK AUTO-GENERATED by Nagios/Icinga RAID event handler

A degraded RAID (broadcom) was detected on host an-worker1199. An automatic snapshot of the current RAID status is attached below.

Please sync with the service owner to find the appropriate time window before actually replacing any failed hardware.

communication: 0 OK : controller: 1 Needs Attention : physical_disk: 2 UGood : virtual_disk: 1 OfLn : bbu: 0 OK : enclosure: 0 OK : CLI Version = 007.1910.0000.0000 Oct 08, 2021

$ sudo /usr/local/lib/nagios/plugins/get-raid-status-broadcom
Failed to execute '['/usr/lib/nagios/plugins/check_nrpe', '-4', '-H', 'an-worker1199', '-c', 'get_raid_status_broadcom']': RETCODE: 2
STDOUT:
communication: 0 OK ; controller: 1 Needs Attention ; physical_disk: 2 UGood ; virtual_disk: 1 OfLn ; bbu: 0 OK ; enclosure: 0 OK ; CLI Version = 007.1910.0000.0000 Oct 08, 2021
Operating system = Linux 5.10.0-34-amd64
Controller = 0
Status = Success
Description = Show Drive Group Succeeded


TOPOLOGY :
========

-----------------------------------------------------------------------------
DG Arr Row EID:Slot DID Type  State BT       Size PDC  PI SED DS3  FSpace TR 
-----------------------------------------------------------------------------
 0 -   -   -        -   RAID1 Optl  N  446.625 GB enbl N  N   dflt N      N  
 0 0   -   -        -   RAID1 Optl  N  446.625 GB enbl N  N   dflt N      N  
 0 0   0   251:0    5   DRIVE Onln  N  446.625 GB enbl N  N   dflt -      N  
 0 0   1   251:1    7   DRIVE Onln  N  446.625 GB enbl N  N   dflt -      N  
 1 -   -   -        -   RAID0 Optl  N    7.276 TB enbl N  N   dflt N      N  
 1 0   -   -        -   RAID0 Optl  N    7.276 TB enbl N  N   dflt N      N  
 1 0   0   252:0    0   DRIVE Onln  N    7.276 TB enbl N  N   dflt -      N  
 2 -   -   -        -   RAID0 Optl  N    7.276 TB enbl N  N   dflt N      N  
 2 0   -   -        -   RAID0 Optl  N    7.276 TB enbl N  N   dflt N      N  
 2 0   0   252:1    6   DRIVE Onln  N    7.276 TB enbl N  N   dflt -      N  
 3 -   -   -        -   RAID0 Optl  N    7.276 TB enbl N  N   dflt N      N  
 3 0   -   -        -   RAID0 Optl  N    7.276 TB enbl N  N   dflt N      N  
 3 0   0   252:3    9   DRIVE Onln  N    7.276 TB enbl N  N   dflt -      N  
 4 -   -   -        -   RAID0 Optl  N    7.276 TB enbl N  N   dflt N      N  
 4 0   -   -        -   RAID0 Optl  N    7.276 TB enbl N  N   dflt N      N  
 4 0   0   252:4    11  DRIVE Onln  N    7.276 TB enbl N  N   dflt -      N  
 5 -   -   -        -   RAID0 Optl  N    7.276 TB enbl N  N   dflt N      N  
 5 0   -   -        -   RAID0 Optl  N    7.276 TB enbl N  N   dflt N      N  
 5 0   0   252:5    10  DRIVE Onln  N    7.276 TB enbl N  N   dflt -      N  
 6 -   -   -        -   RAID0 Optl  N    7.276 TB enbl N  N   dflt N      N  
 6 0   -   -        -   RAID0 Optl  N    7.276 TB enbl N  N   dflt N      N  
 6 0   0   252:6    12  DRIVE Onln  N    7.276 TB enbl N  N   dflt -      N  
 7 -   -   -        -   RAID0 Optl  N    7.276 TB enbl N  N   dflt N      N  
 7 0   -   -        -   RAID0 Optl  N    7.276 TB enbl N  N   dflt N      N  
 7 0   0   252:7    13  DRIVE Onln  N    7.276 TB enbl N  N   dflt -      N  
 8 -   -   -        -   RAID0 Optl  N    7.276 TB enbl N  N   dflt N      N  
 8 0   -   -        -   RAID0 Optl  N    7.276 TB enbl N  N   dflt N      N  
 8 0   0   252:8    1   DRIVE Onln  N    7.276 TB enbl N  N   dflt -      N  
 9 -   -   -        -   RAID0 Optl  N    7.276 TB enbl N  N   dflt N      N  
 9 0   -   -        -   RAID0 Optl  N    7.276 TB enbl N  N   dflt N      N  
 9 0   0   252:9    3   DRIVE Onln  N    7.276 TB enbl N  N   dflt -      N  
10 -   -   -        -   RAID0 OfLn  N    7.276 TB enbl N  N   dflt N      N  
10 0   -   -        -   RAID0 Dgrd  N    7.276 TB enbl N  N   dflt N      N  
11 -   -   -        -   RAID0 Optl  N    7.276 TB enbl N  N   dflt N      N  
11 0   -   -        -   RAID0 Optl  N    7.276 TB enbl N  N   dflt N      N  
11 0   0   252:11   4   DRIVE Onln  N    7.276 TB enbl N  N   dflt -      N  
-----------------------------------------------------------------------------

DG=Disk Group Index|Arr=Array Index|Row=Row Index|EID=Enclosure Device ID
DID=Device ID|Type=Drive Type|Onln=Online|Rbld=Rebuild|Optl=Optimal|Dgrd=Degraded
Pdgd=Partially degraded|Offln=Offline|BT=Background Task Active
PDC=PD Cache|PI=Protection Info|SED=Self Encrypting Drive|Frgn=Foreign
DS3=Dimmer Switch 3|dflt=Default|Msng=Missing|FSpace=Free Space Present
TR=Transport Ready






STDERR:
None

Details

Other Assignee
BTullis

Event Timeline

Dell ticket You have successfully submitted request SR222095997.

Sorry i gave you wrong ticket. thank you will take care of it shortly

Thanks. I've unmounted /dev/sdl1 which was still showing errors on dmesg -T so you can feel free to swap the drive now.
Here is the corrent state of the physical disks.

--------------------------------------------------------------------------------------
EID:Slt DID State  DG       Size Intf Med SED PI SeSz Model                   Sp Type 
--------------------------------------------------------------------------------------
251:0     5 Onln    0 446.625 GB SATA SSD N   N  512B MTFDDAK480TGA-1BC1ZABDA U  -    
251:1     7 Onln    0 446.625 GB SATA SSD N   N  512B MTFDDAK480TGA-1BC1ZABDA U  -    
252:0     0 Onln    1   7.276 TB SATA HDD N   N  512B TOSHIBA MG08ADA800EY    U  -    
252:1     6 Onln    2   7.276 TB SATA HDD N   N  512B TOSHIBA MG08ADA800EY    U  -    
252:2    14 UGood   -   7.276 TB SATA HDD N   N  512B ST8000NM023B-2TJ133     U  -    
252:3     9 Onln    3   7.276 TB SATA HDD N   N  512B TOSHIBA MG08ADA800EY    U  -    
252:4    11 Onln    4   7.276 TB SATA HDD N   N  512B TOSHIBA MG08ADA800EY    U  -    
252:5    10 Onln    5   7.276 TB SATA HDD N   N  512B TOSHIBA MG08ADA800EY    U  -    
252:6    12 Onln    6   7.276 TB SATA HDD N   N  512B TOSHIBA MG08ADA800EY    U  -    
252:7    13 Onln    7   7.276 TB SATA HDD N   N  512B TOSHIBA MG08ADA800EY    U  -    
252:8     1 Onln    8   7.276 TB SATA HDD N   N  512B TOSHIBA MG08ADA800EY    U  -    
252:9     3 Onln    9   7.276 TB SATA HDD N   N  512B TOSHIBA MG08ADA800EY    U  -    
252:10    2 Failed 10   7.276 TB SATA HDD N   N  512B TOSHIBA MG08ADA8        U  -    
252:11    4 Onln   11   7.276 TB SATA HDD N   N  512B TOSHIBA MG08ADA800EY    U  -    
--------------------------------------------------------------------------------------

One Failed and one UGood.

@BTullis @RKemper have you been able to look at this?

Sorry for the delay. I've been looking at this today, as part of updating the Server Profile of all of the an-worker servers.
I should be able to get it properly fixed tomorrow.

This is fixed now.