Page MenuHomePhabricator
Create Task
Maniphest T171183

Degraded RAID on ms-be1016
Closed, ResolvedPublic
Actions

Description

  • Reenable event handler once this is resolved

TASK AUTO-GENERATED by Nagios/Icinga RAID event handler

A degraded RAID (hpssacli) was detected on host ms-be1016. An automatic snapshot of the current RAID status is attached below.

Please sync with the service owner to find the appropriate time window before actually replacing any failed hardware.

CRITICAL: Slot 1: OK: 1I:1:5, 1I:1:6, 1I:1:7, 1I:1:8, 1I:1:1, 1I:1:2, 1I:1:3, 1I:1:4, 2I:2:1, 2I:2:2, 2I:2:3, 2I:2:4, 2I:4:1, 2I:4:2 - Controller: OK - Cache: Permanently Disabled - Cable Error - Battery/Capacitor: Recharging

Smart Array P840 in Slot 1

   array A

      Logical Drive: 1
         Size: 2.7 TB
         Fault Tolerance: 0
         Heads: 255
         Sectors Per Track: 32
         Cylinders: 65535
         Strip Size: 256 KB
         Full Stripe Size: 256 KB
         Status: OK
         Caching:  Enabled
         Unique Identifier: 600508B1001C55499A4F54F7C0145DFD
         Disk Name: /dev/sdc 
         Mount Points: /srv/swift-storage/sdc1 2.7 TB Partition Number 2
         OS Status: LOCKED
         Logical Drive Label: 0076AC50PDNNF0ARH7Z0QDF309
         Drive Type: Data
         LD Acceleration Method: Controller Cache

   array B

      Logical Drive: 2
         Size: 2.7 TB
         Fault Tolerance: 0
         Heads: 255
         Sectors Per Track: 32
         Cylinders: 65535
         Strip Size: 256 KB
         Full Stripe Size: 256 KB
         Status: OK
         Caching:  Enabled
         Unique Identifier: 600508B1001C2655993D3BF1AA2A3E23
         Disk Name: /dev/sdd 
         Mount Points: /srv/swift-storage/sdd1 2.7 TB Partition Number 2
         OS Status: LOCKED
         Logical Drive Label: 0476AC53PDNNF0ARH7Z0QDA548
         Drive Type: Data
         LD Acceleration Method: Controller Cache

   array C

      Logical Drive: 3
         Size: 2.7 TB
         Fault Tolerance: 0
         Heads: 255
         Sectors Per Track: 32
         Cylinders: 65535
         Strip Size: 256 KB
         Full Stripe Size: 256 KB
         Status: OK
         Caching:  Enabled
         Unique Identifier: 600508B1001C566E6026D494123E2B85
         Disk Name: /dev/sde 
         Mount Points: /srv/swift-storage/sde1 2.7 TB Partition Number 2
         OS Status: LOCKED
         Logical Drive Label: 0876AC55PDNNF0ARH7Z0QD52CE
         Drive Type: Data
         LD Acceleration Method: Controller Cache

   array D

      Logical Drive: 4
         Size: 2.7 TB
         Fault Tolerance: 0
         Heads: 255
         Sectors Per Track: 32
         Cylinders: 65535
         Strip Size: 256 KB
         Full Stripe Size: 256 KB
         Status: OK
         Caching:  Enabled
         Unique Identifier: 600508B1001CC13140D79E63A71B8C06
         Disk Name: /dev/sdf 
         Mount Points: /srv/swift-storage/sdf1 2.7 TB Partition Number 2
         OS Status: LOCKED
         Logical Drive Label: 0C76AC57PDNNF0ARH7Z0QD208D
         Drive Type: Data
         LD Acceleration Method: Controller Cache

   array E

      Logical Drive: 5
         Size: 2.7 TB
         Fault Tolerance: 0
         Heads: 255
         Sectors Per Track: 32
         Cylinders: 65535
         Strip Size: 256 KB
         Full Stripe Size: 256 KB
         Status: OK
         Caching:  Enabled
         Unique Identifier: 600508B1001C82E31F9EA24D4799F720
         Disk Name: /dev/sdg 
         Mount Points: /srv/swift-storage/sdg1 2.7 TB Partition Number 2
         OS Status: LOCKED
         Logical Drive Label: 1076AC5APDNNF0ARH7Z0QDF320
         Drive Type: Data
         LD Acceleration Method: Controller Cache

   array F

      Logical Drive: 6
         Size: 2.7 TB
         Fault Tolerance: 0
         Heads: 255
         Sectors Per Track: 32
         Cylinders: 65535
         Strip Size: 256 KB
         Full Stripe Size: 256 KB
         Status: OK
         Caching:  Enabled
         Unique Identifier: 600508B1001C6FCD39772D98C1D34763
         Disk Name: /dev/sdh 
         Mount Points: /srv/swift-storage/sdh1 2.7 TB Partition Number 2
         OS Status: LOCKED
         Logical Drive Label: 1476AC5CPDNNF0ARH7Z0QDDF89
         Drive Type: Data
         LD Acceleration Method: Controller Cache

   array G

      Logical Drive: 7
         Size: 2.7 TB
         Fault Tolerance: 0
         Heads: 255
         Sectors Per Track: 32
         Cylinders: 65535
         Strip Size: 256 KB
         Full Stripe Size: 256 KB
         Status: OK
         Caching:  Enabled
         Unique Identifier: 600508B1001CE9D46F6B652531FC2EB3
         Disk Name: /dev/sdi 
         Mount Points: /srv/swift-storage/sdi1 2.7 TB Partition Number 2
         OS Status: LOCKED
         Logical Drive Label: 1876AC5FPDNNF0ARH7Z0QDB61D
         Drive Type: Data
         LD Acceleration Method: Controller Cache

   array H

      Logical Drive: 8
         Size: 2.7 TB
         Fault Tolerance: 0
         Heads: 255
         Sectors Per Track: 32
         Cylinders: 65535
         Strip Size: 256 KB
         Full Stripe Size: 256 KB
         Status: OK
         Caching:  Enabled
         Unique Identifier: 600508B1001CD65B0FF95079865FB494
         Disk Name: /dev/sdj 
         Mount Points: /srv/swift-storage/sdj1 2.7 TB Partition Number 2
         OS Status: LOCKED
         Logical Drive Label: 1C76AC62PDNNF0ARH7Z0QD9893
         Drive Type: Data
         LD Acceleration Method: Controller Cache

   array I

      Logical Drive: 9
         Size: 2.7 TB
         Fault Tolerance: 0
         Heads: 255
         Sectors Per Track: 32
         Cylinders: 65535
         Strip Size: 256 KB
         Full Stripe Size: 256 KB
         Status: OK
         Caching:  Enabled
         Unique Identifier: 600508B1001C0AF7AD66119F2E11D667
         Disk Name: /dev/sdk 
         Mount Points: /srv/swift-storage/sdk1 2.7 TB Partition Number 2
         OS Status: LOCKED
         Logical Drive Label: 2076AC64PDNNF0ARH7Z0QD7891
         Drive Type: Data
         LD Acceleration Method: Controller Cache

   array J

      Logical Drive: 10
         Size: 2.7 TB
         Fault Tolerance: 0
         Heads: 255
         Sectors Per Track: 32
         Cylinders: 65535
         Strip Size: 256 KB
         Full Stripe Size: 256 KB
         Status: OK
         Caching:  Enabled
         Unique Identifier: 600508B1001C829725CECDA18C05B1E3
         Disk Name: /dev/sdl 
         Mount Points: /srv/swift-storage/sdl1 2.7 TB Partition Number 2
         OS Status: LOCKED
         Logical Drive Label: 2476AC67PDNNF0ARH7Z0QD5BBC
         Drive Type: Data
         LD Acceleration Method: Controller Cache

   array K

      Logical Drive: 11
         Size: 2.7 TB
         Fault Tolerance: 0
         Heads: 255
         Sectors Per Track: 32
         Cylinders: 65535
         Strip Size: 256 KB
         Full Stripe Size: 256 KB
         Status: OK
         Caching:  Enabled
         Unique Identifier: 600508B1001C33D18C7958E96EF64747
         Disk Name: /dev/sdm 
         Mount Points: /srv/swift-storage/sdm1 2.7 TB Partition Number 2
         OS Status: LOCKED
         Logical Drive Label: 2876AC69PDNNF0ARH7Z0QD47F7
         Drive Type: Data
         LD Acceleration Method: Controller Cache

   array L

      Logical Drive: 12
         Size: 2.7 TB
         Fault Tolerance: 0
         Heads: 255
         Sectors Per Track: 32
         Cylinders: 65535
         Strip Size: 256 KB
         Full Stripe Size: 256 KB
         Status: OK
         Caching:  Enabled
         Unique Identifier: 600508B1001CEFA0E54D252A006BE95E
         Disk Name: /dev/sdn 
         Mount Points: /srv/swift-storage/sdn1 2.7 TB Partition Number 2
         OS Status: LOCKED
         Logical Drive Label: 2C76AC6CPDNNF0ARH7Z0QD35D0
         Drive Type: Data
         LD Acceleration Method: Controller Cache

   array M

      Logical Drive: 13
         Size: 279.4 GB
         Fault Tolerance: 0
         Heads: 255
         Sectors Per Track: 32
         Cylinders: 65535
         Strip Size: 256 KB
         Full Stripe Size: 256 KB
         Status: OK
         Caching:  Disabled
         Unique Identifier: 600508B1001CEBDC77F180FF4CB0D535
         Disk Name: /dev/sda 
         Mount Points: /srv/swift-storage/sda4 129.5 GB Partition Number 5, /srv/swift-storage/sda3 93.1 GB Partition Number 4
         OS Status: LOCKED
         Logical Drive Label: 3076AC73PDNNF0ARH7Z0QD70A7
         Drive Type: Data
         LD Acceleration Method: HPE SSD Smart Path

   array N

      Logical Drive: 14
         Size: 279.4 GB
         Fault Tolerance: 0
         Heads: 255
         Sectors Per Track: 32
         Cylinders: 65535
         Strip Size: 256 KB
         Full Stripe Size: 256 KB
         Status: OK
         Caching:  Disabled
         Unique Identifier: 600508B1001C5BA9C48B4EBDA48BBAB9
         Disk Name: /dev/sdb 
         Mount Points: /srv/swift-storage/sdb4 129.5 GB Partition Number 5, /srv/swift-storage/sdb3 93.1 GB Partition Number 4
         OS Status: LOCKED
         Logical Drive Label: 3476AC76PDNNF0ARH7Z0QD4774
         Drive Type: Data
         LD Acceleration Method: HPE SSD Smart Path

Related Objects

Event Timeline

ops-monitoring-bot added projects: SRE, ops-eqiad.Jul 20 2017, 4:00 PM
ops-monitoring-bot subscribed.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 20 2017, 4:00 PM
fgiunchedi assigned this task to • Cmjohnson.Jul 21 2017, 10:13 AM
fgiunchedi triaged this task as Medium priority.
fgiunchedi added subscribers: • Cmjohnson, fgiunchedi.

@Cmjohnson this seems similar to T163777: Debug HP raid cache disabled errors on ms-be1019/20/21 Cache: Permanently Disabled - Cable Error - Battery/Capacitor: Recharging I suspect the machine will need a battery swap

fgiunchedi added a project: User-fgiunchedi.Jul 21 2017, 11:47 AM
fgiunchedi moved this task from Backlog to Other on the User-fgiunchedi board.

Also the controller was swapped not long ago in T150206: ms-be1016 controller cache failure

fgiunchedi merged a task: T171492: Degraded RAID on ms-be1016.Jul 25 2017, 8:28 AM
fgiunchedi moved this task from Backlog to Lower Priority Items on the ops-eqiad board.Jul 25 2017, 9:00 AM
fgiunchedi reopened this task as Open.Jul 25 2017, 1:41 PM
fgiunchedi closed this task as a duplicate of T167264: Degraded RAID on ms-be1016.
fgiunchedi updated the task description. (Show Details)Jul 25 2017, 1:43 PM
• Cmjohnson moved this task from Lower Priority Items to Up next on the ops-eqiad board.Jul 25 2017, 7:09 PM
fgiunchedi raised the priority of this task from Medium to High.Jul 28 2017, 9:36 AM
fgiunchedi mentioned this in T171926: Degraded RAID on ms-be1017.
• Cmjohnson added a comment.Jul 31 2017, 4:18 PM

A ticket has been created with HPE.
Your case was successfully submitted. Please note your Case ID: 5321778513 for future reference.

fgiunchedi added a comment.Aug 2 2017, 3:42 PM

Swapping by @Cmjohnson worked!

root@ms-be1016:~# /usr/local/lib/nagios/plugins/check_hpssacli 
OK: Slot 1: OK: 1I:1:5, 1I:1:6, 1I:1:7, 1I:1:8, 1I:1:1, 1I:1:2, 1I:1:3, 1I:1:4, 2I:2:1, 2I:2:2, 2I:2:3, 2I:2:4, 2I:4:1, 2I:4:2 - Controller: OK - Battery/Capacitor: OK
fgiunchedi updated the task description. (Show Details)Aug 2 2017, 3:50 PM
• Cmjohnson closed this task as Resolved.Aug 2 2017, 4:34 PM

Replaced battery, the old part is not required to be returned.

fgiunchedi mentioned this in T186988: Degraded RAID on ms-be1018.Feb 12 2018, 9:42 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits