Page MenuHomePhabricator

Predictive disk failure on db2047
Closed, ResolvedPublic

Description

Icinga flags a repredictive disk failure on db2047:

WARNING: Slot 0: Predictive Failure: 1I:1:5 - OK: 1I:1:1, 1I:1:2, 1I:1:3, 1I:1:4, 1I:1:6, 1I:1:7, 1I:1:8, 1I:1:9, 1I:1:10, 1I:1:11, 1I:1:12, Controller, Battery/Capacitor

Since the host is still under warranty, I suppose these are eligible for replacement?

Event Timeline

Restricted Application added subscribers: Southparkfan, Aklapper. · View Herald TranscriptNov 1 2016, 8:40 AM
Papaul added a comment.Nov 1 2016, 3:19 PM

@jcrespo can you please give me a detail log on this like you did for T149377?

Thanks.

Papaul triaged this task as Normal priority.Nov 1 2016, 3:29 PM
Papaul added a comment.Nov 1 2016, 8:06 PM

Dear Mr Papaul Tshibamba,

Thank you for contacting Hewlett Packard Enterprise for your service request. This email confirms your request for service and the details are below.

Your request is being worked on under reference number 5314630850
Status: Case is generated and in Progress

Product description: HP ProLiant DL380p Gen8 12 LFF Configure-to-order Server
Product number: 665552-B21
Serial number: 2M245205H6
Subject: DL380p Gen8 - HDD Failure

Yours sincerely,
Hewlett Packard Enterprise

Hey @Papaul
There yo go:

root@db2047:~# hpssacli ctrl all show config

Smart Array P420i in Slot 0 (Embedded)    (sn: 0014380337E0DB0)


   Gen8 ServBP 12+2 at Port 1I, Box 1, OK
   array A (SAS, Unused Space: 0  MB)


      logicaldrive 1 (3.3 TB, RAID 1+0, OK)

      physicaldrive 1I:1:1 (port 1I:box 1:bay 1, SAS, 600 GB, OK)
      physicaldrive 1I:1:2 (port 1I:box 1:bay 2, SAS, 600 GB, OK)
      physicaldrive 1I:1:3 (port 1I:box 1:bay 3, SAS, 600 GB, OK)
      physicaldrive 1I:1:4 (port 1I:box 1:bay 4, SAS, 600 GB, OK)
      physicaldrive 1I:1:5 (port 1I:box 1:bay 5, SAS, 600 GB, Predictive Failure)
      physicaldrive 1I:1:6 (port 1I:box 1:bay 6, SAS, 600 GB, OK)
      physicaldrive 1I:1:7 (port 1I:box 1:bay 7, SAS, 600 GB, OK)
      physicaldrive 1I:1:8 (port 1I:box 1:bay 8, SAS, 600 GB, OK)
      physicaldrive 1I:1:9 (port 1I:box 1:bay 9, SAS, 600 GB, OK)
      physicaldrive 1I:1:10 (port 1I:box 1:bay 10, SAS, 600 GB, OK)
      physicaldrive 1I:1:11 (port 1I:box 1:bay 11, SAS, 600 GB, OK)
      physicaldrive 1I:1:12 (port 1I:box 1:bay 12, SAS, 600 GB, OK)

   Enclosure SEP (Vendor ID HP, Model Gen8 ServBP 12+2) 378  (WWID: 50014380324C6459, Port: 1I, Box: 1)

   Expander 380  (WWID: 50014380324C6440, Port: 1I, Box: 1)

   SEP (Vendor ID PMCSIERA, Model SRCv8x6G) 379  (WWID: 50014380337E0DBF)
Papaul reassigned this task from Papaul to Marostegui.Nov 2 2016, 6:19 PM
Papaul added a subscriber: Papaul.

Disk placement complete.

Marostegui closed this task as Resolved.Nov 3 2016, 9:40 AM

This is now good:

root@db2047:~# hpssacli ctrl all show config

Smart Array P420i in Slot 0 (Embedded)    (sn: 0014380337E0DB0)


   Gen8 ServBP 12+2 at Port 1I, Box 1, OK
   array A (SAS, Unused Space: 0  MB)


      logicaldrive 1 (3.3 TB, RAID 1+0, OK)

      physicaldrive 1I:1:1 (port 1I:box 1:bay 1, SAS, 600 GB, OK)
      physicaldrive 1I:1:2 (port 1I:box 1:bay 2, SAS, 600 GB, OK)
      physicaldrive 1I:1:3 (port 1I:box 1:bay 3, SAS, 600 GB, OK)
      physicaldrive 1I:1:4 (port 1I:box 1:bay 4, SAS, 600 GB, OK)
      physicaldrive 1I:1:5 (port 1I:box 1:bay 5, SAS, 600 GB, OK)
      physicaldrive 1I:1:6 (port 1I:box 1:bay 6, SAS, 600 GB, OK)
      physicaldrive 1I:1:7 (port 1I:box 1:bay 7, SAS, 600 GB, OK)
      physicaldrive 1I:1:8 (port 1I:box 1:bay 8, SAS, 600 GB, OK)
      physicaldrive 1I:1:9 (port 1I:box 1:bay 9, SAS, 600 GB, OK)
      physicaldrive 1I:1:10 (port 1I:box 1:bay 10, SAS, 600 GB, OK)
      physicaldrive 1I:1:11 (port 1I:box 1:bay 11, SAS, 600 GB, OK)
      physicaldrive 1I:1:12 (port 1I:box 1:bay 12, SAS, 600 GB, OK)

   Enclosure SEP (Vendor ID HP, Model Gen8 ServBP 12+2) 378  (WWID: 50014380324C6459, Port: 1I, Box: 1)

   Expander 380  (WWID: 50014380324C6440, Port: 1I, Box: 1)

   SEP (Vendor ID PMCSIERA, Model SRCv8x6G) 379  (WWID: 50014380337E0DBF)
Dzahn reopened this task as Open.Apr 18 2019, 9:10 PM
Dzahn added a subscriber: Dzahn.

https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=db2047&service=Device+not+healthy+-SMART-

Service
Device not healthy -SMART-
On Host
db2047

*cluster=mysql device=cciss,11 instance=db2047:9100 job=node site=codfw*

Dzahn added a comment.Apr 18 2019, 9:11 PM
@db2047:~# hpssacli ctrl all show config

Smart Array P420i in Slot 0 (Embedded)    (sn: 0014380337E0DB0)


   Port Name: 1I

   Port Name: 2I

   Gen8 ServBP 12+2 at Port 1I, Box 1, OK
   array A (SAS, Unused Space: 0  MB)


      logicaldrive 1 (3.3 TB, RAID 1+0, OK)

      physicaldrive 1I:1:1 (port 1I:box 1:bay 1, SAS, 600 GB, Predictive Failure)
      physicaldrive 1I:1:2 (port 1I:box 1:bay 2, SAS, 600 GB, OK)
      physicaldrive 1I:1:3 (port 1I:box 1:bay 3, SAS, 600 GB, OK)
      physicaldrive 1I:1:4 (port 1I:box 1:bay 4, SAS, 600 GB, OK)
      physicaldrive 1I:1:5 (port 1I:box 1:bay 5, SAS, 600 GB, OK)
      physicaldrive 1I:1:6 (port 1I:box 1:bay 6, SAS, 600 GB, OK)
      physicaldrive 1I:1:7 (port 1I:box 1:bay 7, SAS, 600 GB, OK)
      physicaldrive 1I:1:8 (port 1I:box 1:bay 8, SAS, 600 GB, OK)
      physicaldrive 1I:1:9 (port 1I:box 1:bay 9, SAS, 600 GB, OK)
      physicaldrive 1I:1:10 (port 1I:box 1:bay 10, SAS, 600 GB, OK)
      physicaldrive 1I:1:11 (port 1I:box 1:bay 11, SAS, 600 GB, OK)
      physicaldrive 1I:1:12 (port 1I:box 1:bay 12, SAS, 600 GB, Predictive Failure)

   Enclosure SEP (Vendor ID HP, Model Gen8 ServBP 12+2) 378  (WWID: 50014380324C6459, Port: 1I, Box: 1)

   Expander 380  (WWID: 50014380324C6440, Port: 1I, Box: 1)

   SEP (Vendor ID PMCSIERA, Model SRCv8x6G) 379  (WWID: 50014380337E0DBF)
Marostegui closed this task as Resolved.Apr 19 2019, 6:03 AM

Thanks! We are tracking those at T208323 and as we have many - we are waiting for them to fully fail before replacing (as sometimes it takes months) so closing this again as it is on the other task and an automatic task will be created once the disk is fully gailed.
Thanks for letting us know though, much appreciated!

(also not the same disk slot, so different issues and should be tracked separately)

Dzahn removed a subscriber: Dzahn.Apr 23 2019, 10:25 PM