Degraded RAID on kubestage2004
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	ops-monitoring-bot
	Fri, Feb 6, 6:04 PM

Description

TASK AUTO-GENERATED by Nagios/Icinga RAID event handler

A degraded RAID (md) was detected on host kubestage2004. An automatic snapshot of the current RAID status is attached below.

Please sync with the service owner to find the appropriate time window before actually replacing any failed hardware.

CRITICAL: State: degraded, Active: 1, Working: 1, Failed: 1, Spare: 0

$ sudo /usr/local/lib/nagios/plugins/get-raid-status-md
Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5] [raid4] [raid10] 
md0 : active raid1 sdb2[1](F) sda2[0]
      937267200 blocks super 1.2 [2/1] [U_]
      bitmap: 3/7 pages [12KB], 65536KB chunk

unused devices: <none>

Related Objects

Mentioned Here: T393205: Degraded RAID on kubestage2004

Event Timeline

ops-monitoring-bot created this task.Fri, Feb 6, 6:04 PM

Restricted Application added a project: DC-Ops. · View Herald TranscriptFri, Feb 6, 6:04 PM

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

Jhancock.wm moved this task from Backlog to Hardware Failure / Troubleshoot on the ops-codfw board.Mon, Feb 9, 3:53 PM

already troubleshoot this back in May T393205. still under warranty, starting a replacement ticket with Dell: SR222489236

@JMeybohm Dell shipped the drive. should arrive late today or tomorrow. all good for me to replace the drive?

@Jhancock.wm yeah, anytime. Thanks!

for some reason the package still hasn't been delivered. looping back to dell to figure out why

The tracking for the shipment now says they were unable to deliver the package and are sending to back to the shipper. I have asked Dell to resend.

new part shipped last night

Jhancock.wm merged a task: T418027: Degraded RAID on kubestage2004.Mon, Feb 23, 3:58 PM

@JMeybohm disk has been replaced.

Rebuild started, thanks!

Personalities : [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md0 : active raid1 sda2[2] sdb2[0]
      937262528 blocks super 1.2 [2/1] [U_]
      [>....................]  recovery =  0.0% (102016/937262528) finish=612.3min speed=25504K/sec
      
unused devices: <none>

Degraded RAID on kubestage2004Closed, ResolvedPublicActions

Description

Related Objects

Event Timeline

Degraded RAID on kubestage2004
Closed, ResolvedPublic
Actions