Page MenuHomePhabricator
Create Task
Maniphest T282839

Degraded RAID on ms-be1053
Closed, ResolvedPublic
Actions

Description

TASK AUTO-GENERATED by Nagios/Icinga RAID event handler

A degraded RAID (ssacli) was detected on host ms-be1053. An automatic snapshot of the current RAID status is attached below.

Please sync with the service owner to find the appropriate time window before actually replacing any failed hardware.

CRITICAL: Slot 0: Failed: 1I:1:2 - OK: 1I:1:1, 1I:1:3, 1I:1:4, 2I:2:1, 2I:2:2, 2I:2:3, 2I:2:4, 3I:3:1, 3I:3:2, 3I:3:3, 3I:3:4, 4I:5:1, 4I:5:2 - Controller: OK - Battery/Capacitor: OK

$ sudo /usr/local/lib/nagios/plugins/get-raid-status-ssacli

HPE Smart Array P816i-a SR Gen10 in Slot 0 (Embedded)

   Array A

      Logical Drive: 1
         Size: 447.10 GB
         Fault Tolerance: 0
         Strip Size: 256 KB
         Full Stripe Size: 256 KB
         Status: OK
         MultiDomain Status: OK
         Caching:  Disabled
         Disk Name: /dev/sda 
         Mount Points: /srv/swift-storage/sda4 297.2 GB Partition Number 4, /srv/swift-storage/sda3 93.1 GB Partition Number 3
         OS Status: LOCKED
         Boot Volume: Primary
         Drive Type: Data
         LD Acceleration Method: Smart Path


   Array B

      Logical Drive: 2
         Size: 447.10 GB
         Fault Tolerance: 0
         Strip Size: 256 KB
         Full Stripe Size: 256 KB
         Status: OK
         MultiDomain Status: OK
         Caching:  Disabled
         Disk Name: /dev/sdb 
         Mount Points: /srv/swift-storage/sdb4 297.2 GB Partition Number 4, /srv/swift-storage/sdb3 93.1 GB Partition Number 3
         OS Status: LOCKED
         Boot Volume: Secondary
         Drive Type: Data
         LD Acceleration Method: Smart Path


   Array C

      Logical Drive: 3
         Size: 3.64 TB
         Fault Tolerance: 0
         Strip Size: 256 KB
         Full Stripe Size: 256 KB
         Status: OK
         MultiDomain Status: OK
         Caching:  Enabled
         Disk Name: /dev/sdc 
         Mount Points: /srv/swift-storage/sdc1 3.6 TB Partition Number 1
         OS Status: LOCKED
         Drive Type: Data
         LD Acceleration Method: Controller Cache


   Array D

      Logical Drive: 4
         Size: 3.64 TB
         Fault Tolerance: 0
         Strip Size: 256 KB
         Full Stripe Size: 256 KB
         Status: Failed
         MultiDomain Status: OK
         Caching:  Enabled
         Drive Type: Data
         LD Acceleration Method: Controller Cache


   Array E

      Logical Drive: 5
         Size: 3.64 TB
         Fault Tolerance: 0
         Strip Size: 256 KB
         Full Stripe Size: 256 KB
         Status: OK
         MultiDomain Status: OK
         Caching:  Enabled
         Disk Name: /dev/sde 
         Mount Points: /srv/swift-storage/sde1 3.6 TB Partition Number 1
         OS Status: LOCKED
         Drive Type: Data
         LD Acceleration Method: Controller Cache


   Array F

      Logical Drive: 6
         Size: 3.64 TB
         Fault Tolerance: 0
         Strip Size: 256 KB
         Full Stripe Size: 256 KB
         Status: OK
         MultiDomain Status: OK
         Caching:  Enabled
         Disk Name: /dev/sdf 
         Mount Points: /srv/swift-storage/sdf1 3.6 TB Partition Number 1
         OS Status: LOCKED
         Drive Type: Data
         LD Acceleration Method: Controller Cache


   Array G

      Logical Drive: 7
         Size: 3.64 TB
         Fault Tolerance: 0
         Strip Size: 256 KB
         Full Stripe Size: 256 KB
         Status: OK
         MultiDomain Status: OK
         Caching:  Enabled
         Disk Name: /dev/sdg 
         Mount Points: /srv/swift-storage/sdg1 3.6 TB Partition Number 1
         OS Status: LOCKED
         Drive Type: Data
         LD Acceleration Method: Controller Cache


   Array H

      Logical Drive: 8
         Size: 3.64 TB
         Fault Tolerance: 0
         Strip Size: 256 KB
         Full Stripe Size: 256 KB
         Status: OK
         MultiDomain Status: OK
         Caching:  Enabled
         Disk Name: /dev/sdh 
         Mount Points: /srv/swift-storage/sdh1 3.6 TB Partition Number 1
         OS Status: LOCKED
         Drive Type: Data
         LD Acceleration Method: Controller Cache


   Array I

      Logical Drive: 9
         Size: 3.64 TB
         Fault Tolerance: 0
         Strip Size: 256 KB
         Full Stripe Size: 256 KB
         Status: OK
         MultiDomain Status: OK
         Caching:  Enabled
         Disk Name: /dev/sdi 
         Mount Points: /srv/swift-storage/sdi1 3.6 TB Partition Number 1
         OS Status: LOCKED
         Drive Type: Data
         LD Acceleration Method: Controller Cache


   Array J

      Logical Drive: 10
         Size: 3.64 TB
         Fault Tolerance: 0
         Strip Size: 256 KB
         Full Stripe Size: 256 KB
         Status: OK
         MultiDomain Status: OK
         Caching:  Enabled
         Disk Name: /dev/sdj 
         Mount Points: /srv/swift-storage/sdj1 3.6 TB Partition Number 1
         OS Status: LOCKED
         Drive Type: Data
         LD Acceleration Method: Controller Cache


   Array K

      Logical Drive: 11
         Size: 3.64 TB
         Fault Tolerance: 0
         Strip Size: 256 KB
         Full Stripe Size: 256 KB
         Status: OK
         MultiDomain Status: OK
         Caching:  Enabled
         Disk Name: /dev/sdk 
         Mount Points: /srv/swift-storage/sdk1 3.6 TB Partition Number 1
         OS Status: LOCKED
         Drive Type: Data
         LD Acceleration Method: Controller Cache


   Array L

      Logical Drive: 12
         Size: 3.64 TB
         Fault Tolerance: 0
         Strip Size: 256 KB
         Full Stripe Size: 256 KB
         Status: OK
         MultiDomain Status: OK
         Caching:  Enabled
         Disk Name: /dev/sdl 
         Mount Points: /srv/swift-storage/sdl1 3.6 TB Partition Number 1
         OS Status: LOCKED
         Drive Type: Data
         LD Acceleration Method: Controller Cache


   Array M

      Logical Drive: 13
         Size: 3.64 TB
         Fault Tolerance: 0
         Strip Size: 256 KB
         Full Stripe Size: 256 KB
         Status: OK
         MultiDomain Status: OK
         Caching:  Enabled
         Disk Name: /dev/sdm 
         Mount Points: /srv/swift-storage/sdm1 3.6 TB Partition Number 1
         OS Status: LOCKED
         Drive Type: Data
         LD Acceleration Method: Controller Cache


   Array N

      Logical Drive: 14
         Size: 3.64 TB
         Fault Tolerance: 0
         Strip Size: 256 KB
         Full Stripe Size: 256 KB
         Status: OK
         MultiDomain Status: OK
         Caching:  Enabled
         Disk Name: /dev/sdn 
         Mount Points: /srv/swift-storage/sdn1 3.6 TB Partition Number 1
         OS Status: LOCKED
         Drive Type: Data
         LD Acceleration Method: Controller Cache

Event Timeline

ops-monitoring-bot created this task.May 13 2021, 10:17 PM
fgiunchedi added subscribers: Jclark-ctr, • Cmjohnson, fgiunchedi.May 14 2021, 9:06 AM

sdd is indeed busted and host is under warranty, please replace @Cmjohnson / @Jclark-ctr , thank you!

RhinosF1 subscribed.May 14 2021, 2:52 PM
wiki_willy assigned this task to • Cmjohnson.May 14 2021, 9:37 PM
• Cmjohnson added a comment.May 20 2021, 3:16 PM

Thank you, I will get a ticket in with HPE ASAP

• Cmjohnson moved this task from Backlog to Hardware Failure / Troubleshoot on the ops-eqiad board.May 20 2021, 3:17 PM
• Cmjohnson added a comment.May 25 2021, 5:59 PM

The case was submitted with HPE,
Successfully Submitted Case Number: 5355909720

• Cmjohnson added a comment.May 27 2021, 4:04 PM

HPE is sending the part, they sent me an email requesting duplicate information that I missed. Taken care of and the part should be here tomorrow or Tuesday.

• Cmjohnson closed this task as Resolved.Jun 3 2021, 5:51 PM

replaced the disk

fgiunchedi added a comment.Jun 4 2021, 8:36 AM

Disk is rebuilding, thank you @Cmjohnson

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL