Page MenuHomePhabricator
Create Task
Maniphest T228606

Degraded RAID on elastic1046
Closed, ResolvedPublic
Actions

Description

TASK AUTO-GENERATED by Nagios/Icinga RAID event handler

A degraded RAID (md) was detected on host elastic1046. An automatic snapshot of the current RAID status is attached below.

Please sync with the service owner to find the appropriate time window before actually replacing any failed hardware.

CRITICAL: State: degraded, Active: 3, Working: 3, Failed: 1, Spare: 0

$ sudo /usr/local/lib/nagios/plugins/get-raid-status-md
Personalities : [raid1] [raid0] [linear] [multipath] [raid6] [raid5] [raid4] [raid10] 
md1 : active raid0 sda2[0] sdb2[1]
      1503967232 blocks super 1.2 512k chunks
      
md0 : active raid1 sda1[0] sdb1[1](F)
      29279232 blocks super 1.2 [2/1] [U_]
      
unused devices: <none>

Details

ProjectBranchLines +/-Subject
operations/puppetproduction+3 -0elasticsearch: ensure python prometheus client in installed
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedJclark-ctrT228606 Degraded RAID on elastic1046
Unknown Object (Task)
Unknown Object (Task)

Event Timeline

ops-monitoring-bot added projects: SRE, ops-eqiad.Jul 21 2019, 8:10 PM
ops-monitoring-bot added a subscriber: ops-monitoring-bot.
Stashbot added a comment.Jul 22 2019, 7:54 AM

Mentioned in SAL (#wikimedia-operations) [2019-07-22T07:54:48Z] <elukey> sudo -i depool on elastic1046 - broken disk (srv partition not available) - T228606

elukey added a subscriber: elukey.Jul 22 2019, 7:55 AM
elukey@elastic1046:~$ sudo -i depool
Depooling all services on elastic1046.eqiad.wmnet
eqiad/elasticsearch/elasticsearch/elastic1046.eqiad.wmnet: pooled changed yes => no
eqiad/elasticsearch/elasticsearch-psi-ssl/elastic1046.eqiad.wmnet: pooled changed yes => no
eqiad/elasticsearch/elasticsearch-ssl/elastic1046.eqiad.wmnet: pooled changed yes => no
elukey triaged this task as Medium priority.Jul 22 2019, 8:05 AM
Cmjohnson added a subscriber: Cmjohnson.Jul 22 2019, 2:33 PM
This comment was removed by Cmjohnson.
Cmjohnson moved this task from Backlog to Hardware Failure / Troubleshoot on the ops-eqiad board.Jul 22 2019, 2:34 PM
wiki_willy assigned this task to Cmjohnson.Jul 22 2019, 11:05 PM
Volans added a subscriber: Gehel.Jul 23 2019, 6:43 AM
Cmjohnson reassigned this task from Cmjohnson to wiki_willy.Jul 24 2019, 5:39 PM
Cmjohnson added a subscriber: wiki_willy.

This server is out of warranty, ended April 2019. @wiki_willy escalating to you to decide on disks

Cmjohnson moved this task from Hardware Failure / Troubleshoot to Blocked on the ops-eqiad board.Jul 24 2019, 5:39 PM
wiki_willy added a comment.Jul 24 2019, 6:21 PM

@elukey - since elastic1046 is just barely out of warranty (only by a few months), we'll still have to purchase a new disk for this server. Just double-checking that's the route you want to go, before we place the order.

Thanks,
Willy

elukey added a subscriber: dcausse.Jul 25 2019, 8:25 AM

Adding @dcausse to the conversation since @Gehel is on holiday. I would simply buy the disk now, but not sure if elastic1046 is scheduled to be refreshed soon.

elukey added a comment.Jul 25 2019, 8:31 AM

The host is not scheduled for replacement, @wiki_willy please proceed with the order of the disk :)

jijiki added a subscriber: jijiki.Jul 25 2019, 11:56 AM
wiki_willy reassigned this task from wiki_willy to Cmjohnson.Jul 25 2019, 4:10 PM

Thanks @elukey, subtask #T229017 has been opened to order the replacement drive with procurement. Assigning this task back to @Cmjohnson, for when the disk arrives onsite.

Thanks,
Willy

wiki_willy added a comment.Aug 20 2019, 6:47 PM

Confirmed by Chris that the drive arrived on August 8

Cmjohnson added a comment.Aug 23 2019, 3:30 PM

I replaced the failed disk

Gehel claimed this task.Aug 23 2019, 3:38 PM

@Cmjohnson thanks! I'll take it over and reimage

ops-monitoring-bot added a comment.Aug 27 2019, 12:19 PM

Script wmf-auto-reimage was launched by gehel on cumin1001.eqiad.wmnet for hosts:

['elastic1046.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201908271218_gehel_128010.log.

Gehel added a project: Discovery-Search (Current work).Aug 27 2019, 12:19 PM
ops-monitoring-bot added a comment.Aug 27 2019, 12:53 PM

Completed auto-reimage of hosts:

['elastic1046.eqiad.wmnet']

Of which those FAILED:

['elastic1046.eqiad.wmnet']
Gehel added a comment.Aug 27 2019, 12:56 PM

@Cmjohnson: it looks like the installer only sees a single disk, and thus can't partition. Could you check? Thanks!

Gehel reassigned this task from Gehel to Cmjohnson.Aug 27 2019, 12:56 PM
wiki_willy added a comment.Sep 10 2019, 6:29 PM

@Cmjohnson - could be the drive is seated securely or possibly a loose cable /connection

jijiki removed a subscriber: jijiki.Sep 11 2019, 12:44 PM
Cmjohnson added a comment.Sep 12 2019, 3:47 PM

@wiki_willy not really but I reseated it anyway. As far as I can tell in bios everything looks normal. I did swap the 2 disks. @Gehel try again please.

Cmjohnson added a comment.Sep 12 2019, 3:50 PM

I did notice that ssds are different types
The new ssd is a DC3320 series
The old ssd is a DC3610 series

ops-monitoring-bot added a comment.Sep 13 2019, 7:42 AM

Script wmf-auto-reimage was launched by gehel on cumin1001.eqiad.wmnet for hosts:

['elastic1046.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201909130742_gehel_73868.log.

ops-monitoring-bot added a comment.Sep 13 2019, 8:47 AM

Completed auto-reimage of hosts:

['elastic1046.eqiad.wmnet']

Of which those FAILED:

['elastic1046.eqiad.wmnet']
ops-monitoring-bot added a comment.Sep 18 2019, 9:37 AM

Script wmf-auto-reimage was launched by gehel on cumin1001.eqiad.wmnet for hosts:

['elastic1046.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201909180937_gehel_148677.log.

ops-monitoring-bot added a comment.Sep 18 2019, 10:43 AM

Completed auto-reimage of hosts:

['elastic1046.eqiad.wmnet']

Of which those FAILED:

['elastic1046.eqiad.wmnet']
Gehel added a comment.Sep 18 2019, 12:02 PM

Tried again, seems that sda has issues (see log below). Is it that the second disk also failed? Or that the wrong disk was replaced? Or something else?

@Cmjohnson any idea what to do at this point?

Sep 18 09:41:13 kernel: [   50.172541] ata1: EH complete
Sep 18 09:41:13 kernel: [   50.172569] Dev sda: unable to read RDB block 0
Sep 18 09:41:13 kernel: [   50.183837] ata1.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
Sep 18 09:41:13 kernel: [   50.183840] ata1.00: irq_stat 0x40000001
Sep 18 09:41:13 kernel: [   50.183843] ata1.00: failed command: READ DMA
Sep 18 09:41:13 kernel: [   50.183853] ata1.00: cmd c8/00:08:00:00:00/00:00:00:00:00/e0 tag 13 dma 4096 in
Sep 18 09:41:13 kernel: [   50.183853]          res 53/10:00:00:00:00/00:00:00:00:00/00 Emask 0x81 (invalid argument)
Sep 18 09:41:13 kernel: [   50.183855] ata1.00: status: { DRDY SENSE ERR }
Sep 18 09:41:13 kernel: [   50.183857] ata1.00: error: { IDNF }
Sep 18 09:41:13 kernel: [   50.184508] ata1.00: configured for UDMA/100
Sep 18 09:41:13 kernel: [   50.184520] ata1: EH complete
Sep 18 09:41:13 kernel: [   50.195837] ata1.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
Sep 18 09:41:13 kernel: [   50.195840] ata1.00: irq_stat 0x40000001
Sep 18 09:41:13 kernel: [   50.195844] ata1.00: failed command: READ DMA
wiki_willy added a comment.Oct 7 2019, 9:16 PM

@Cmjohnson - let me know if we need to order a replacement drive (along with what type of disk), since it's out of warranty. Thanks, Willy

Cmjohnson added a comment.Oct 22 2019, 12:19 PM

@wiki_willy can you order a new disk? I tried logging in but being prompted for a password so I cannot get disk info.

Cmjohnson reassigned this task from Cmjohnson to wiki_willy.Oct 22 2019, 12:19 PM
wiki_willy added a subtask: Unknown Object (Task).Oct 22 2019, 4:10 PM

Procurement task created for Rob to order replacement drive. Thanks, Willy

wiki_willy reassigned this task from wiki_willy to Jclark-ctr.Oct 22 2019, 4:15 PM
Gehel moved this task from Incoming to Blocked/Waiting on the Discovery-Search (Current work) board.Oct 28 2019, 4:24 PM
Jclark-ctr added a comment.Nov 4 2019, 10:01 PM

received replacement drive @elukey @Gehel.

Jclark-ctr added a comment.Nov 4 2019, 10:17 PM

Replaced drive sda . @Gehel

ops-monitoring-bot added a comment.Nov 4 2019, 10:49 PM

Script wmf-auto-reimage was launched by gehel on cumin1001.eqiad.wmnet for hosts:

['elastic1046.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201911042249_gehel_234323.log.

ops-monitoring-bot added a comment.Nov 4 2019, 11:12 PM

Completed auto-reimage of hosts:

['elastic1046.eqiad.wmnet']

and were ALL successful.

Gehel added a comment.Nov 4 2019, 11:17 PM

Host reimage, joined the cluster, repooled, no errors.

Netbox updated to reflect the host being active again.

It looks like we're all good!

Gehel moved this task from Blocked/Waiting to Needs Reporting on the Discovery-Search (Current work) board.Nov 4 2019, 11:17 PM
wiki_willy added a comment.Nov 4 2019, 11:18 PM

Thanks @Gehel , thanks @Jclark-ctr - I'll go ahead and resolve this task. Thanks, Willy

wiki_willy closed this task as Resolved.Nov 4 2019, 11:18 PM
wiki_willy closed subtask Unknown Object (Task) as Resolved.Nov 4 2019, 11:21 PM
wiki_willy mentioned this in Unknown Object (Task).
gerritbot added a comment.Nov 4 2019, 11:29 PM

Change 548548 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/puppet@production] elasticsearch: ensure python prometheus client in installed

https://gerrit.wikimedia.org/r/548548

gerritbot added a project: Patch-For-Review.Nov 4 2019, 11:29 PM
gerritbot added a comment.Nov 4 2019, 11:34 PM

Change 548548 merged by Gehel:
[operations/puppet@production] elasticsearch: ensure python prometheus client in installed

https://gerrit.wikimedia.org/r/548548

Jclark-ctr closed subtask Unknown Object (Task) as Resolved.Nov 13 2019, 4:01 PM
Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL