Page MenuHomePhabricator

Degraded RAID on cloudvirt1024
Closed, DuplicatePublic

Description

TASK AUTO-GENERATED by Nagios/Icinga RAID event handler

A degraded RAID (megacli) was detected on host cloudvirt1024. An automatic snapshot of the current RAID status is attached below.

Please sync with the service owner to find the appropriate time window before actually replacing any failed hardware.

CRITICAL: 1 failed LD(s) (Degraded)

$ sudo /usr/local/lib/nagios/plugins/get-raid-status-megacli
=== RaidStatus (does not include components in optimal state)
name: Adapter #0

	Virtual Drive: 0 (Target Id: 0)
	RAID Level: Primary-1, Secondary-0, RAID Level Qualifier-0
	State: =====> Degraded <=====
	Number Of Drives: 8
	Number of Spans: 1
	Current Cache Policy: WriteBack, ReadAhead, Direct, No Write Cache if Bad BBU

		Span: 0 - Number of PDs: 8

			PD: 0 Information
			Enclosure Device ID: 32
			Slot Number: 8
			Drive's position: DiskGroup: 0, Span: 0, Arm: 0
			Media Error Count: 0
			Other Error Count: 467
			Predictive Failure Count: 0
			Last Predictive Failure Event Seq Number: 0

				Raw Size: 1.746 TB [0xdf8fe2b0 Sectors]
				Firmware state: =====> Rebuild <=====
				Media Type: Solid State Device
				Drive Temperature: 25C (77.00 F)

=== RaidStatus completed

Event Timeline

aborrero triaged this task as High priority.Sep 27 2019, 8:45 AM

Apparently, updating the firmware didn't do it. This system has something wrong with it's RAID system that likes to fail out lots of disks at a time (occasionally "fixing" them with a reboot). T215892 T230442 T230289
Then there's https://wikitech.wikimedia.org/wiki/Incident_documentation/20190213-cloudvps

I think this needs to get merged into T230289