Page MenuHomePhabricator

Degraded RAID on db1103
Closed, ResolvedPublic

Description

TASK AUTO-GENERATED by Nagios/Icinga RAID event handler

A degraded RAID (megacli) was detected on host db1103. An automatic snapshot of the current RAID status is attached below.

Please sync with the service owner to find the appropriate time window before actually replacing any failed hardware.

CRITICAL: 1 failed LD(s) (Degraded)

$ sudo /usr/local/lib/nagios/plugins/get-raid-status-megacli
=== RaidStatus (does not include components in optimal state)
name: Adapter #0

	Virtual Drive: 0 (Target Id: 0)
	RAID Level: Primary-1, Secondary-0, RAID Level Qualifier-0
	State: =====> Degraded <=====
	Number Of Drives: 10
	Number of Spans: 1
	Current Cache Policy: WriteBack, ReadAhead, Direct, No Write Cache if Bad BBU

		Span: 0 - Number of PDs: 10

			PD: 2 Information
			ERROR: =====> MISSING DRIVE INFO <=====

=== RaidStatus completed

Event Timeline

Restricted Application added a subscriber: Marostegui. · View Herald TranscriptSat, Feb 20, 9:13 AM
Marostegui triaged this task as High priority.Sat, Feb 20, 1:15 PM
Marostegui added a project: DBA.
Marostegui added subscribers: wiki_willy, LSobanski.

This is X1 primary master, @wiki_willy can we give it some priority? Thanks

Marostegui moved this task from Triage to In progress on the DBA board.Sat, Feb 20, 1:15 PM

Ack @Marostegui, we'll take a look at it, with whoever heads onsite first this week. @Cmjohnson or @Jclark-ctr - since this machine is out of warranty, can you see if you can grab a spare drive from one of the decom'd servers? Thanks, Willy

This is X1 primary master, @wiki_willy can we give it some priority? Thanks

@Marostegui Swapped Bad SSD @wiki_willy we did have one new in box same size same model ect. it originally came from HP

Thanks @Jclark-ctr

@Marostegui Swapped Bad SSD @wiki_willy we did have one new in box same size same model ect. it originally came from HP

Cmjohnson closed this task as Resolved.Tue, Feb 23, 3:23 PM

This appears to have been done by @Jclark-ctr

Thank you for the fast response.
Confirming this is all good now:

root@db1103:~# megacli -LDInfo -Lall -aALL


Adapter 0 -- Virtual Drive Information:
Virtual Drive: 0 (Target Id: 0)
Name                :
RAID Level          : Primary-1, Secondary-0, RAID Level Qualifier-0
Size                : 3.635 TB
Sector Size         : 512
Is VD emulated      : Yes
Mirror Data         : 3.635 TB
State               : Optimal
Strip Size          : 256 KB
Number Of Drives    : 10
Span Depth          : 1
Default Cache Policy: WriteBack, ReadAhead, Direct, No Write Cache if Bad BBU
Current Cache Policy: WriteBack, ReadAhead, Direct, No Write Cache if Bad BBU
Default Access Policy: Read/Write
Current Access Policy: Read/Write
Disk Cache Policy   : Disk's Default
Encryption Type     : None
Default Power Savings Policy: Controller Defined
Current Power Savings Policy: None
Can spin up in 1 minute: No
LD has drives that support T10 power conditions: No
LD's IO profile supports MAX power savings with cached writes: No
Bad Blocks Exist: No
Is VD Cached: No