Page MenuHomePhabricator
Create Task
Maniphest T191720

Degraded RAID on db2069
Closed, ResolvedPublic
Actions

Description

TASK AUTO-GENERATED by Nagios/Icinga RAID event handler

A degraded RAID (hpssacli) was detected on host db2069. An automatic snapshot of the current RAID status is attached below.

Please sync with the service owner to find the appropriate time window before actually replacing any failed hardware.

CRITICAL: Slot 0: Predictive Failure: 1I:1:1 - Failed: 1I:1:4 - OK: 1I:1:2, 1I:1:3, 1I:1:5, 1I:1:6, 1I:1:7, 1I:1:8, 1I:1:9, 1I:1:10, 1I:1:11, 1I:1:12 - Controller: OK - Battery/Capacitor: OK

$ sudo /usr/local/lib/nagios/plugins/get_raid_status_hpssacli

Smart Array P420i in Slot 0 (Embedded)

   array A

      Logical Drive: 1
         Size: 3.3 TB
         Fault Tolerance: 1+0
         Strip Size: 256 KB
         Full Stripe Size: 1536 KB
         Status: Interim Recovery Mode
         Caching:  Enabled
         Disk Name: /dev/sda 
         Mount Points: / 37.3 GB Partition Number 2
         OS Status: LOCKED
         Mirror Group 1:
            physicaldrive 1I:1:1 (port 1I:box 1:bay 1, SAS, 600 GB, Predictive Failure)
            physicaldrive 1I:1:2 (port 1I:box 1:bay 2, SAS, 600 GB, OK)
            physicaldrive 1I:1:3 (port 1I:box 1:bay 3, SAS, 600 GB, OK)
            physicaldrive 1I:1:4 (port 1I:box 1:bay 4, SAS, 600 GB, Failed)
            physicaldrive 1I:1:5 (port 1I:box 1:bay 5, SAS, 600 GB, OK)
            physicaldrive 1I:1:6 (port 1I:box 1:bay 6, SAS, 600 GB, OK)
         Mirror Group 2:
            physicaldrive 1I:1:7 (port 1I:box 1:bay 7, SAS, 600 GB, OK)
            physicaldrive 1I:1:8 (port 1I:box 1:bay 8, SAS, 600 GB, OK)
            physicaldrive 1I:1:9 (port 1I:box 1:bay 9, SAS, 600 GB, OK)
            physicaldrive 1I:1:10 (port 1I:box 1:bay 10, SAS, 600 GB, OK)
            physicaldrive 1I:1:11 (port 1I:box 1:bay 11, SAS, 600 GB, OK)
            physicaldrive 1I:1:12 (port 1I:box 1:bay 12, SAS, 600 GB, OK)
         Drive Type: Data
         LD Acceleration Method: Controller Cache

Related Objects

Event Timeline

ops-monitoring-bot added projects: ops-codfw, SRE.Apr 8 2018, 12:36 PM
ops-monitoring-bot subscribed.
Restricted Application added subscribers: • Marostegui, Aklapper. · View Herald TranscriptApr 8 2018, 12:36 PM
• Marostegui assigned this task to Papaul.Apr 8 2018, 1:31 PM
• Marostegui added a project: DBA.

Hi!
Please get the disk changed when you can.
Thanks!

• Marostegui mentioned this in T191593: db2069 RAID with predictive failure.Apr 8 2018, 1:49 PM
• Marostegui moved this task from Triage to In progress on the DBA board.Apr 9 2018, 5:49 AM
Papaul triaged this task as Medium priority.Apr 9 2018, 2:54 PM
Papaul added a comment.Apr 9 2018, 2:58 PM

@Marostegui which disk you want to replace first? 1 or 4 ?

• Marostegui added a comment.Apr 9 2018, 2:58 PM

Oh, I didn't know there were two of them broken. Let me check

• Marostegui added a comment.Apr 9 2018, 2:59 PM

@Papaul disk #4

Papaul reassigned this task from Papaul to • Marostegui.Apr 9 2018, 3:05 PM
Papaul subscribed.

complete

• Marostegui added a comment.Apr 9 2018, 3:08 PM

Thanks

physicaldrive 1I:1:4 (port 1I:box 1:bay 4, SAS, 600 GB, Rebuilding)

Once this is completed we can replace #1

• Marostegui reassigned this task from • Marostegui to Papaul.Apr 9 2018, 5:57 PM

@Papaul please change disk #1 as disk #4 got rebuilt finely

logicaldrive 1 (3.3 TB, RAID 1+0, OK)

physicaldrive 1I:1:1 (port 1I:box 1:bay 1, SAS, 600 GB, Predictive Failure)
physicaldrive 1I:1:2 (port 1I:box 1:bay 2, SAS, 600 GB, OK)
physicaldrive 1I:1:3 (port 1I:box 1:bay 3, SAS, 600 GB, OK)
physicaldrive 1I:1:4 (port 1I:box 1:bay 4, SAS, 600 GB, OK)
physicaldrive 1I:1:5 (port 1I:box 1:bay 5, SAS, 600 GB, OK)
physicaldrive 1I:1:6 (port 1I:box 1:bay 6, SAS, 600 GB, OK)
physicaldrive 1I:1:7 (port 1I:box 1:bay 7, SAS, 600 GB, OK)
physicaldrive 1I:1:8 (port 1I:box 1:bay 8, SAS, 600 GB, OK)
physicaldrive 1I:1:9 (port 1I:box 1:bay 9, SAS, 600 GB, OK)
physicaldrive 1I:1:10 (port 1I:box 1:bay 10, SAS, 600 GB, OK)
physicaldrive 1I:1:11 (port 1I:box 1:bay 11, SAS, 600 GB, OK)
physicaldrive 1I:1:12 (port 1I:box 1:bay 12, SAS, 600 GB, OK)
• Marostegui merged a task: T191826: Degraded RAID on db2069.Apr 9 2018, 7:03 PM

I see the disk already being rebuilt
Thanks @Papaul

physicaldrive 1I:1:1 (port 1I:box 1:bay 1, SAS, 600 GB, Rebuilding)
Papaul added a comment.Apr 9 2018, 7:13 PM

@Marostegui no problem

• Marostegui added a comment.Apr 9 2018, 8:16 PM

@Papaul this disk failed, was it an used one?
Can you try another one?

physicaldrive 1I:1:1 (port 1I:box 1:bay 1, SAS, 600 GB, Failed)

Thanks

• Marostegui merged a task: T191836: Degraded RAID on db2069.Apr 9 2018, 8:17 PM
Papaul added a comment.Apr 9 2018, 8:21 PM

Ok will do once back at the DC in the AM

• Marostegui added a comment.Apr 9 2018, 8:27 PM

Thanks!

Papaul reassigned this task from Papaul to • Marostegui.Apr 10 2018, 2:55 PM

Disk replacement complete

• Marostegui added a comment.Apr 10 2018, 2:56 PM

Let's hope this time it finishes correctly!

logicaldrive 1 (3.3 TB, RAID 1+0, Recovering, 1% complete)

physicaldrive 1I:1:1 (port 1I:box 1:bay 1, SAS, 600 GB, Rebuilding)

Thanks!

• Marostegui closed this task as Resolved.Apr 10 2018, 5:49 PM

This is all good now!
Thanks

logicaldrive 1 (3.3 TB, RAID 1+0, OK)

physicaldrive 1I:1:1 (port 1I:box 1:bay 1, SAS, 600 GB, OK)
physicaldrive 1I:1:2 (port 1I:box 1:bay 2, SAS, 600 GB, OK)
physicaldrive 1I:1:3 (port 1I:box 1:bay 3, SAS, 600 GB, OK)
physicaldrive 1I:1:4 (port 1I:box 1:bay 4, SAS, 600 GB, OK)
physicaldrive 1I:1:5 (port 1I:box 1:bay 5, SAS, 600 GB, OK)
physicaldrive 1I:1:6 (port 1I:box 1:bay 6, SAS, 600 GB, OK)
physicaldrive 1I:1:7 (port 1I:box 1:bay 7, SAS, 600 GB, OK)
physicaldrive 1I:1:8 (port 1I:box 1:bay 8, SAS, 600 GB, OK)
physicaldrive 1I:1:9 (port 1I:box 1:bay 9, SAS, 600 GB, OK)
physicaldrive 1I:1:10 (port 1I:box 1:bay 10, SAS, 600 GB, OK)
physicaldrive 1I:1:11 (port 1I:box 1:bay 11, SAS, 600 GB, OK)
physicaldrive 1I:1:12 (port 1I:box 1:bay 12, SAS, 600 GB, OK)
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL