Page MenuHomePhabricator

Degraded RAID on an-worker1200
Closed, ResolvedPublic

Description

TASK AUTO-GENERATED by Nagios/Icinga RAID event handler

A degraded RAID (broadcom) was detected on host an-worker1200. An automatic snapshot of the current RAID status is attached below.

Please sync with the service owner to find the appropriate time window before actually replacing any failed hardware.

communication: 0 OK : controller: 1 Needs Attention : physical_disk: 2 UGood : virtual_disk: 1 OfLn : bbu: 0 OK : enclosure: 0 OK : CLI Version = 007.1910.0000.0000 Oct 08, 2021

$ sudo /usr/local/lib/nagios/plugins/get-raid-status-broadcom
Failed to execute '['/usr/lib/nagios/plugins/check_nrpe', '-4', '-H', 'an-worker1200', '-c', 'get_raid_status_broadcom']': RETCODE: 2
STDOUT:
communication: 0 OK ; controller: 1 Needs Attention ; physical_disk: 2 UGood ; virtual_disk: 1 OfLn ; bbu: 0 OK ; enclosure: 0 OK ; CLI Version = 007.1910.0000.0000 Oct 08, 2021
Operating system = Linux 6.1.0-52-amd64
Controller = 0
Status = Success
Description = Show Drive Group Succeeded


TOPOLOGY :
========

-----------------------------------------------------------------------------
DG Arr Row EID:Slot DID Type  State BT       Size PDC  PI SED DS3  FSpace TR 
-----------------------------------------------------------------------------
 0 -   -   -        -   RAID1 Optl  N  446.625 GB enbl N  N   dflt N      N  
 0 0   -   -        -   RAID1 Optl  N  446.625 GB enbl N  N   dflt N      N  
 0 0   0   251:0    12  DRIVE Onln  N  446.625 GB enbl N  N   dflt -      N  
 0 0   1   251:1    13  DRIVE Onln  N  446.625 GB enbl N  N   dflt -      N  
 1 -   -   -        -   RAID0 Optl  N    7.276 TB enbl N  N   dflt N      N  
 1 0   -   -        -   RAID0 Optl  N    7.276 TB enbl N  N   dflt N      N  
 1 0   0   252:0    0   DRIVE Onln  N    7.276 TB enbl N  N   dflt -      N  
 2 -   -   -        -   RAID0 Optl  N    7.276 TB enbl N  N   dflt N      N  
 2 0   -   -        -   RAID0 Optl  N    7.276 TB enbl N  N   dflt N      N  
 2 0   0   252:1    2   DRIVE Onln  N    7.276 TB enbl N  N   dflt -      N  
 3 -   -   -        -   RAID0 Optl  N    7.276 TB enbl N  N   dflt N      N  
 3 0   -   -        -   RAID0 Optl  N    7.276 TB enbl N  N   dflt N      N  
 3 0   0   252:2    3   DRIVE Onln  N    7.276 TB enbl N  N   dflt -      N  
 4 -   -   -        -   RAID0 Optl  N    7.276 TB enbl N  N   dflt N      N  
 4 0   -   -        -   RAID0 Optl  N    7.276 TB enbl N  N   dflt N      N  
 4 0   0   252:3    4   DRIVE Onln  N    7.276 TB enbl N  N   dflt -      N  
 5 -   -   -        -   RAID0 Optl  N    7.276 TB enbl N  N   dflt N      N  
 5 0   -   -        -   RAID0 Optl  N    7.276 TB enbl N  N   dflt N      N  
 5 0   0   252:4    8   DRIVE Onln  N    7.276 TB enbl N  N   dflt -      N  
 6 -   -   -        -   RAID0 Optl  N    7.276 TB enbl N  N   dflt N      N  
 6 0   -   -        -   RAID0 Optl  N    7.276 TB enbl N  N   dflt N      N  
 6 0   0   252:5    5   DRIVE Onln  N    7.276 TB enbl N  N   dflt -      N  
 7 -   -   -        -   RAID0 Optl  N    7.276 TB enbl N  N   dflt N      N  
 7 0   -   -        -   RAID0 Optl  N    7.276 TB enbl N  N   dflt N      N  
 7 0   0   252:6    6   DRIVE Onln  N    7.276 TB enbl N  N   dflt -      N  
 8 -   -   -        -   RAID0 Optl  N    7.276 TB enbl N  N   dflt N      N  
 8 0   -   -        -   RAID0 Optl  N    7.276 TB enbl N  N   dflt N      N  
 8 0   0   252:9    9   DRIVE Onln  N    7.276 TB enbl N  N   dflt -      N  
 9 -   -   -        -   RAID0 Optl  N    7.276 TB enbl N  N   dflt N      N  
 9 0   -   -        -   RAID0 Optl  N    7.276 TB enbl N  N   dflt N      N  
 9 0   0   252:10   10  DRIVE Onln  N    7.276 TB enbl N  N   dflt -      N  
10 -   -   -        -   RAID0 Optl  N    7.276 TB enbl N  N   dflt N      N  
10 0   -   -        -   RAID0 Optl  N    7.276 TB enbl N  N   dflt N      N  
10 0   0   252:11   11  DRIVE Onln  N    7.276 TB enbl N  N   dflt -      N  
11 -   -   -        -   RAID0 OfLn  N    7.276 TB dflt N  N   dflt N      N  
11 0   -   -        -   RAID0 Dgrd  N    7.276 TB dflt N  N   dflt N      N  
-----------------------------------------------------------------------------

DG=Disk Group Index|Arr=Array Index|Row=Row Index|EID=Enclosure Device ID
DID=Device ID|Type=Drive Type|Onln=Online|Rbld=Rebuild|Optl=Optimal|Dgrd=Degraded
Pdgd=Partially degraded|Offln=Offline|BT=Background Task Active
PDC=PD Cache|PI=Protection Info|SED=Self Encrypting Drive|Frgn=Foreign
DS3=Dimmer Switch 3|dflt=Default|Msng=Missing|FSpace=Free Space Present
TR=Transport Ready






STDERR:
None

Details

Other Assignee
BTullis

Event Timeline

Jclark-ctr updated Other Assignee, added: BTullis.
Jclark-ctr subscribed.

Dell ticket opened SR231432096
Parts Arrive Wednesday, Sep 9, 5:00 p.m.

Parts have not arrived yet will follow up tomorrow with dell

This one keeps dropping disks. I've added the parent task where I'm doing a larger investigation into what seems like a batch of servers where this keeps happening.

I think that this is is another one of those times when the disk hasn't actually failed, but it was disconnected by the RAID controller, temporarily.

I went to upgrade the firmware for the server from the management web UI and it was showing a failed drive.

On reboot, the controller detected this as a 'foreign configuration'.
Then I imported the foreign configuration and now everything is back as it should be, without having swapped the disk.

I should have thought to take screenshots of each step, but I only got one at the end.

image.png (1,935×813 px, 194 KB)

I don't know what you want to do about the replacement parts comin from Dell for this.

Returning drive to dell will request replacement again if fails again