(OoW) Degraded RAID on es2003
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	ops-monitoring-bot
	Jun 5 2019, 6:08 PM

Description

TASK AUTO-GENERATED by Nagios/Icinga RAID event handler

A degraded RAID (megacli) was detected on host es2003. An automatic snapshot of the current RAID status is attached below.

Please sync with the service owner to find the appropriate time window before actually replacing any failed hardware.

CRITICAL: 1 failed LD(s) (Degraded)

$ sudo /usr/local/lib/nagios/plugins/get-raid-status-megacli
=== RaidStatus (does not include components in optimal state)
name: Adapter #0

	Virtual Drive: 0 (Target Id: 0)
	RAID Level: Primary-1, Secondary-0, RAID Level Qualifier-0
	State: =====> Degraded <=====
	Number Of Drives per span: 2
	Number of Spans: 6
	Current Cache Policy: WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU

		Span: 5 - Number of PDs: 2

			PD: 0 Information
			Enclosure Device ID: 32
			Slot Number: 10
			Drive's position: DiskGroup: 0, Span: 5, Arm: 0
			Media Error Count: 0
			Other Error Count: 1
			Predictive Failure Count: 0
			Last Predictive Failure Event Seq Number: 0

				Raw Size: 1.819 TB [0xe8e088b0 Sectors]
				Firmware state: =====> Failed <=====
				Media Type: Hard Disk Device
				Drive Temperature: N/A

=== RaidStatus completed

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Resolved		Papaul	T222592 Decommission es2001, es2002, es2003, es2004
		Resolved		jcrespo	T225131 (OoW) Degraded RAID on es2003

Event Timeline

ops-monitoring-bot added projects: ops-codfw, SRE.Jun 5 2019, 6:08 PM

ops-monitoring-bot subscribed.

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 5 2019, 6:08 PM

@jcrespo I guess the transfer put some unexpected stress on these disks

RobH moved this task from Backlog to Hardware Failure / Troubleshoot on the ops-codfw board.Jun 12 2019, 1:10 PM

ArielGlenn triaged this task as Medium priority.Jun 13 2019, 7:23 AM

Marostegui added a parent task: T222592: Decommission es2001, es2002, es2003, es2004.Jun 13 2019, 7:41 AM

wiki_willy renamed this task from Degraded RAID on es2003 to (OoW) Degraded RAID on es2003.Jul 15 2019, 7:43 PM

@Marostegui @jcrespo can you tell if it is 2TB SATA or SAS? IF it is 2TB SATA we have some new onces onsite.

SAS HD disks of 1.819 TB.

Disk replaced

All good - thanks!

root@es2003:/usr/local/lib/nagios/plugins# megacli -LDPDInfo -aAll

Adapter #0

Number of Virtual Disks: 1
Virtual Drive: 0 (Target Id: 0)
Name                :allr10
RAID Level          : Primary-1, Secondary-0, RAID Level Qualifier-0
Size                : 10.913 TB
Sector Size         : 512
Mirror Data         : 10.913 TB
State               : Optimal
Strip Size          : 256 KB
Number Of Drives per span:2
Span Depth          : 6
Default Cache Policy: WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU
Current Cache Policy: WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU
Default Access Policy: Read/Write
Current Access Policy: Read/Write
Disk Cache Policy   : Disk's Default
Encryption Type     : None
Bad Blocks Exist: Yes
Is VD Cached: Yes
Cache Cade Type : Read Only
Number of Spans: 6
Span: 0 - Number of PDs: 2

(OoW) Degraded RAID on es2003Closed, ResolvedPublicActions

Description

Related ObjectsSearch...

Event Timeline

(OoW) Degraded RAID on es2003
Closed, ResolvedPublic
Actions

Related Objects
Search...