Page MenuHomePhabricator
Create Task
Maniphest T244986

cloudvirt1009: Device not healthy -SMART-
Closed, ResolvedPublic
Actions

Description

@MoritzMuehlenhoff reported we have an icinga alert for cloudvirt1009 related to the disk.

image.png (659×730 px, 57 KB)

However, I checked this:

aborrero@cloudvirt1009:~ $ sudo smartctl -H /dev/sdb
smartctl 6.6 2016-05-31 r4324 [x86_64-linux-4.9.0-11-amd64] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF READ SMART DATA SECTION ===
SMART Health Status: OK

aborrero@cloudvirt1009:~ $ sudo smartctl -H /dev/sda
smartctl 6.6 2016-05-31 r4324 [x86_64-linux-4.9.0-11-amd64] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF READ SMART DATA SECTION ===
SMART Health Status: OK

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedJclark-ctrT244986 cloudvirt1009: Device not healthy -SMART-
Unknown Object (Task)

Event Timeline

• aborrero created this task.Feb 12 2020, 11:22 AM
Restricted Application added a project: SRE. · View Herald TranscriptFeb 12 2020, 11:22 AM
• aborrero closed this task as Resolved.Feb 12 2020, 11:41 AM
• aborrero claimed this task.

Apparently the issue resolved itself:

image.png (733×725 px, 63 KB)

Volans reopened this task as Open.Feb 12 2020, 2:29 PM
Volans subscribed.

Re-opening as it's currently alerting.

It's all documented in the link that is associated with the alert itself:
https://wikitech.wikimedia.org/wiki/SMART#Alerts

From the Icinga alert pick the disk (cciss,17):

cluster=wmcs device=cciss,17 instance=cloudvirt1009:9100 job=node site=eqiad

Then run smart-data-dump and see how the affected disk is referred to:

DEBUG:__main__:Running: /usr/bin/timeout 60 /usr/sbin/smartctl --info --health -d cciss,17 /dev/sda
DEBUG:__main__:Running: /usr/bin/timeout 60 /usr/sbin/smartctl --attributes -d cciss,17 /dev/sda

So then you can run /usr/sbin/smartctl --info --health -d cciss,17 /dev/sda that at the end has:

=== START OF READ SMART DATA SECTION ===
SMART Health Status: SPINDLE IMPENDING FAILURE GENERAL HARD DRIVE FAILURE [asc=5d, ascq=50]

And also see the attributes with the other command if you're interested.

• aborrero mentioned this in T198479: labvirt1009 HP Raid alert.Feb 12 2020, 4:20 PM
• aborrero reassigned this task from • aborrero to Jclark-ctr.Feb 13 2020, 5:16 PM
• aborrero added subscribers: Jclark-ctr, • Cmjohnson.

General hard drive failure doesn't sound good. Please @Jclark-ctr @Cmjohnson advice how to proceed.

wiki_willy moved this task from Backlog to Cloud Tasks on the ops-eqiad board.Feb 18 2020, 9:24 PM
Jclark-ctr added a subscriber: wiki_willy.Feb 27 2020, 12:27 AM

@wiki_willy I have checked our storage room we have no spares host is 5 years old at the time drive needed is a 300gb 15k sas. current drive installed part number is 748385

wiki_willy added a comment.Feb 27 2020, 12:50 AM

@aborrero (and @Jclark-ctr for visibility) - it looks this was purchased back in 2014, and past the 5yr server life cycle. Would it be possible to decommission this host instead? Let us know if there's a corresponding install task, that we can help with, in getting this one refreshed. Thanks, Willy

bd808 added a comment.Feb 27 2020, 2:20 PM
In T244986#5922092, @wiki_willy wrote:

@aborrero (and @Jclark-ctr for visibility) - it looks this was purchased back in 2014, and past the 5yr server life cycle. Would it be possible to decommission this host instead? Let us know if there's a corresponding install task, that we can help with, in getting this one refreshed. Thanks, Willy

@wiki_willy {T243471} is related to replacement. The cloudvirt1001 - cloudvirt1009 hosts are all due for refresh in Q4 according to the hardware procurement spreadsheet. The task I linked is us trying to get a few of those done sooner exactly because we have so many active hardware issue right now. We are in a "fun" spot where these disk tickets are getting pushback because the hosts are old while at the same time the procurement for replacements is de-prioritized because it is considered an out of order request.

wiki_willy added a comment.Feb 27 2020, 5:19 PM

@bd808 - thanks for providing the background context around these. I hit up Rob to prioritize T243471 more. (quotes being submitted soon) Also, we'll go ahead and order a replacement disk for cloudvirt1009 as well. Thanks, Willy

wiki_willy added a subtask: Unknown Object (Task).Feb 27 2020, 5:24 PM

T246365 created for ordering the replacement drive. Thanks, Willy

Stashbot added a comment.Feb 29 2020, 4:32 PM

Mentioned in SAL (#wikimedia-cloud) [2020-02-29T16:32:12Z] <bstorm_> downtimed the smart alert on cloudvirt1009 until Monday since apparently predictive failures flap T244986

Jclark-ctr closed subtask Unknown Object (Task) as Resolved.Mar 10 2020, 8:57 PM
Jclark-ctr added a comment.Mar 11 2020, 8:42 PM

replaced drive slot 17, removed 2nd failed drive in slot 18

• JHedden added a comment.Mar 11 2020, 8:53 PM

Cleaned up the RAID config with hpssacli ctrl slot=0 array b remove spares=2I:1:18

• JHedden closed this task as Resolved.Mar 11 2020, 8:53 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits