Degraded RAID on ms-be2018
Closed, InvalidPublic
Actions

Assigned To

None

Authored By

	ops-monitoring-bot
	Dec 22 2018, 1:35 PM

Description

TASK AUTO-GENERATED by Nagios/Icinga RAID event handler

A degraded RAID (md) was detected on host ms-be2018. An automatic snapshot of the current RAID status is attached below.

Please sync with the service owner to find the appropriate time window before actually replacing any failed hardware.

CRITICAL: State: degraded, Active: 3, Working: 3, Failed: 1, Spare: 0

$ sudo /usr/local/lib/nagios/plugins/get_raid_status_md
Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5] [raid4] [raid10] 
md1 : active (auto-read-only) raid1 sdb2[1] sda2[0]
      976320 blocks super 1.2 [2/2] [UU]
      	resync=PENDING
      
md0 : active raid1 sdb1[1] sda1[0](F)
      58559488 blocks super 1.2 [2/1] [_U]
      
unused devices: <none>

Event Timeline

ops-monitoring-bot added projects: ops-codfw, SRE.Dec 22 2018, 1:35 PM

ops-monitoring-bot subscribed.

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptDec 22 2018, 1:35 PM

Host's raid controller locked up, a reboot brought things back. I'll update the controller firmware after the holidays JIC.

ms-be2018:~$ cat /proc/mdstat 
Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5] [raid4] [raid10] 
md1 : active (auto-read-only) raid1 sdb2[1] sda2[0]
      976320 blocks super 1.2 [2/2] [UU]
      	resync=PENDING
      
md0 : active raid1 sda1[0] sdb1[1]
      58559488 blocks super 1.2 [2/2] [UU]
      
unused devices: <none>

Degraded RAID on ms-be2018Closed, InvalidPublicActions

Description

Event Timeline

Degraded RAID on ms-be2018
Closed, InvalidPublic
Actions