Degraded RAID on db2047
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	ops-monitoring-bot
	Jan 4 2019, 7:37 PM

Description

TASK AUTO-GENERATED by Nagios/Icinga RAID event handler

A degraded RAID (hpssacli) was detected on host db2047. An automatic snapshot of the current RAID status is attached below.

Please sync with the service owner to find the appropriate time window before actually replacing any failed hardware.

CRITICAL: Slot 0: OK: 1I:1:2, 1I:1:3, 1I:1:4, 1I:1:5, 1I:1:6, 1I:1:7, 1I:1:8, 1I:1:9, 1I:1:10, 1I:1:11, 1I:1:12 - Failed: 1I:1:1 - Controller: OK - Battery/Capacitor: OK

$ sudo /usr/local/lib/nagios/plugins/get_raid_status_hpssacli

Smart Array P420i in Slot 0 (Embedded)

   array A

      Logical Drive: 1
         Size: 3.3 TB
         Fault Tolerance: 1+0
         Strip Size: 256 KB
         Full Stripe Size: 1536 KB
         Status: Interim Recovery Mode
         Caching:  Enabled
         Disk Name: /dev/sda 
         Mount Points: / 37.3 GB Partition Number 2
         OS Status: LOCKED
         Mirror Group 1:
            physicaldrive 1I:1:1 (port 1I:box 1:bay 1, SAS, 600 GB, Failed)
            physicaldrive 1I:1:2 (port 1I:box 1:bay 2, SAS, 600 GB, OK)
            physicaldrive 1I:1:3 (port 1I:box 1:bay 3, SAS, 600 GB, OK)
            physicaldrive 1I:1:4 (port 1I:box 1:bay 4, SAS, 600 GB, OK)
            physicaldrive 1I:1:5 (port 1I:box 1:bay 5, SAS, 600 GB, OK)
            physicaldrive 1I:1:6 (port 1I:box 1:bay 6, SAS, 600 GB, OK)
         Mirror Group 2:
            physicaldrive 1I:1:7 (port 1I:box 1:bay 7, SAS, 600 GB, OK)
            physicaldrive 1I:1:8 (port 1I:box 1:bay 8, SAS, 600 GB, OK)
            physicaldrive 1I:1:9 (port 1I:box 1:bay 9, SAS, 600 GB, OK)
            physicaldrive 1I:1:10 (port 1I:box 1:bay 10, SAS, 600 GB, OK)
            physicaldrive 1I:1:11 (port 1I:box 1:bay 11, SAS, 600 GB, OK)
            physicaldrive 1I:1:12 (port 1I:box 1:bay 12, SAS, 600 GB, OK)
         Drive Type: Data
         LD Acceleration Method: Controller Cache

Related Objects

Mentioned In: T212969: Degraded RAID alert not acking notifications
T208323: Predictive failures on disk S.M.A.R.T. status

Event Timeline

ops-monitoring-bot added projects: ops-codfw, SRE.Jan 4 2019, 7:37 PM

ops-monitoring-bot subscribed.

Restricted Application added subscribers: • Banyek, • Marostegui, Aklapper. · View Herald TranscriptJan 4 2019, 7:37 PM

Let's get it replaced when you can
Thanks!

• Marostegui triaged this task as Medium priority.Jan 4 2019, 7:42 PM

• Marostegui moved this task from Triage to In progress on the DBA board.

• Marostegui mentioned this in T208323: Predictive failures on disk S.M.A.R.T. status.Jan 4 2019, 7:44 PM

• Marostegui mentioned this in T212969: Degraded RAID alert not acking notifications.Jan 4 2019, 8:02 PM

Disk replacement complete

Thanks!

root@db2047:~# hpssacli controller all show config

Smart Array P420i in Slot 0 (Embedded)    (sn: 0014380337E0DB0)


   Port Name: 1I

   Port Name: 2I

   Gen8 ServBP 12+2 at Port 1I, Box 1, OK
   array A (SAS, Unused Space: 0  MB)


      logicaldrive 1 (3.3 TB, RAID 1+0, OK)

Degraded RAID on db2047Closed, ResolvedPublicActions

Description

Related Objects

Event Timeline

Degraded RAID on db2047
Closed, ResolvedPublic
Actions