Page MenuHomePhabricator
Create Task
Maniphest T220907

Degraded RAID on ms-be1013
Closed, ResolvedPublic
Actions

Description

TASK AUTO-GENERATED by Nagios/Icinga RAID event handler

A degraded RAID (md) was detected on host ms-be1013. An automatic snapshot of the current RAID status is attached below.

Please sync with the service owner to find the appropriate time window before actually replacing any failed hardware.

CRITICAL: State: degraded, Active: 3, Working: 3, Failed: 1, Spare: 0

$ sudo /usr/local/lib/nagios/plugins/get-raid-status-md
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md1 : active (auto-read-only) raid1 sdm2[0] sdn2[1]
      976320 blocks super 1.2 [2/2] [UU]
      
md0 : active raid1 sdm1[0](F) sdn1[1]
      29279232 blocks super 1.2 [2/1] [_U]
      
unused devices: <none>

Related Objects

Event Timeline

ops-monitoring-bot added projects: SRE, ops-eqiad.Apr 14 2019, 10:38 AM
ops-monitoring-bot subscribed.
Krenair merged a task: T220910: Degraded RAID on ms-be1013.Apr 14 2019, 2:35 PM
fgiunchedi subscribed.Apr 15 2019, 2:00 PM

Tried to reboot the host in the hope the controller freaked out and a reboot would "fix" it or at least reset. However the host isn't coming back, and console says No more sessions are available for this type of connection!. An mc reset cold as per https://wikitech.wikimedia.org/wiki/Platform-specific_documentation/Dell_PowerEdge_RN20_Gen8#Troubleshooting also didn't seem to help.

At any rate, I've started the decom of this host as per T220590: Decom ms-be101[345] as we need to do that regardless.

colewhite triaged this task as High priority.Apr 16 2019, 3:42 PM
• Cmjohnson subscribed.Apr 17 2019, 8:37 PM

@fgiunchedi do you want to power off unplug and power on...that will clear the issue

fgiunchedi added a comment.Apr 23 2019, 12:18 PM
In T220907#5120604, @Cmjohnson wrote:

@fgiunchedi do you want to power off unplug and power on...that will clear the issue

Yes please drain the power! Thanks

Dzahn moved this task from Backlog to Hardware Failure / Troubleshoot on the ops-eqiad board.Apr 30 2019, 6:34 PM
fgiunchedi added a comment.May 2 2019, 9:08 AM
In T220907#5130265, @fgiunchedi wrote:
In T220907#5120604, @Cmjohnson wrote:

@fgiunchedi do you want to power off unplug and power on...that will clear the issue

Yes please drain the power! Thanks

Before draining the power please unplug the production network or disable this host interface on the switch.

This host has been offline for a while now and we are decomissioning it anyways, I suspect if we bring it back on the network now it'll start pushing/replicating its outdated versions of objects. Best to keep it disconnected from the network for now.

• Cmjohnson changed the task status from Open to Stalled.May 7 2019, 2:47 PM
• Cmjohnson moved this task from Hardware Failure / Troubleshoot to Stalled on the ops-eqiad board.

I drained the flea power and you should not have any issues with the idrac. The server is still out of warranty so not much I can do about the raid at this point in time. I did disabled the network port switch. If you need access again please let me know and I will enable the port.

Stalling on my end for now.

fgiunchedi closed this task as Resolved.May 23 2019, 10:02 AM
fgiunchedi claimed this task.

I'm resolving this since we're going to decom this host in T220907: Degraded RAID on ms-be1013, thanks @Cmjohnson !

fgiunchedi mentioned this in T220590: Decom ms-be101[345].May 23 2019, 10:03 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL