Page MenuHomePhabricator
Create Task
Maniphest T230088

cloudelastic1002: SMART/disk error
Closed, ResolvedPublic
Actions

Description

There's some issue around cloudelastic1002 /dev/sdb device according to icinga: https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=cloudelastic1002&service=Device+not+healthy+-SMART-

onimisionipe@cloudelastic1002:~$ sudo smartctl -H /dev/sdb
smartctl 6.6 2016-05-31 r4324 [x86_64-linux-4.9.0-9-amd64] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: FAILED!
Drive failure expected in less than 24 hours. SAVE ALL DATA.
Failed Attributes:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
201 Power_Loss_Cap_Test     0x0033   001   001   010    Pre-fail  Always   FAILING_NOW 4 (82 6532)

Related Objects

Event Timeline

• Mathew.onipe triaged this task as Medium priority.Aug 8 2019, 2:52 AM
• Mathew.onipe created this task.
wiki_willy assigned this task to Cmjohnson.Aug 8 2019, 9:59 PM
Cmjohnson moved this task from Backlog to Cloud Tasks on the ops-eqiad board.Aug 15 2019, 3:03 PM
• bd808 moved this task from Inbox to Doing on the cloud-services-team (Kanban) board.Sep 11 2019, 3:13 PM
• bd808 moved this task from Doing to Watching on the cloud-services-team (Kanban) board.Sep 12 2019, 9:29 PM
• JHedden added a subscriber: • JHedden.Dec 3 2019, 10:31 PM
Jclark-ctr added a subscriber: Jclark-ctr.Dec 3 2019, 10:35 PM

Confirmed: Service Request 1005040764 was successfully submitted. ordered replacement drive

Jclark-ctr added a comment.Dec 5 2019, 6:52 PM

@mathew.onip Disk arrived

elukey added subscribers: aborrero, elukey.Dec 5 2019, 7:23 PM

@Mathew.onipe @Gehel @aborrero can you sync with @Jclark-ctr when you have a minute to swap the disk?

Stashbot added a comment.Dec 5 2019, 8:11 PM

Mentioned in SAL (#wikimedia-operations) [2019-12-05T20:11:32Z] <onimisionipe> ban cloudelastic1002 from shard allocation - T230088

Jclark-ctr added a comment.Dec 5 2019, 10:20 PM

replaced failed drive

• Mathew.onipe added a comment.Dec 5 2019, 10:32 PM
onimisionipe@cloudelastic1002:~$ sudo smartctl -H /dev/sdb
smartctl 6.6 2016-05-31 r4324 [x86_64-linux-4.9.0-9-amd64] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

It looks good now. Thanks @Jclark-ctr

wiki_willy mentioned this in T239957: Degraded RAID on cloudelastic1002.Dec 5 2019, 11:16 PM
• Mathew.onipe closed this task as Resolved.Dec 6 2019, 9:07 AM
Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL