Degraded RAID on kafka-jumbo1001
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	ops-monitoring-bot
	May 1 2020, 12:56 AM

Description

TASK AUTO-GENERATED by Nagios/Icinga RAID event handler

A degraded RAID (megacli) was detected on host kafka-jumbo1001. An automatic snapshot of the current RAID status is attached below.

Please sync with the service owner to find the appropriate time window before actually replacing any failed hardware.

CRITICAL: 1 failed LD(s) (Degraded)

$ sudo /usr/local/lib/nagios/plugins/get-raid-status-megacli
=== RaidStatus (does not include components in optimal state)
name: Adapter #0

	Virtual Drive: 1 (Target Id: 1)
	RAID Level: Primary-1, Secondary-0, RAID Level Qualifier-0
	State: =====> Degraded <=====
	Number Of Drives: 12
	Number of Spans: 1
	Current Cache Policy: WriteBack, ReadAhead, Direct, No Write Cache if Bad BBU

		Span: 0 - Number of PDs: 12

			PD: 8 Information
			ERROR: =====> MISSING DRIVE INFO <=====

=== RaidStatus completed

Event Timeline

ops-monitoring-bot created this task.May 1 2020, 12:56 AM

Looks like the warranty on kafka-jumbo1001 is going to end in a few weeks. @Jclark-ctr or @Cmjohnson - can one of you guys troubleshoot and submit the RMA for this part before then? Thanks, Willy

wiki_willy assigned this task to • Cmjohnson.May 4 2020, 7:55 PM

@elukey I need to power this server off, I am not able to reach the mgmt/idrac for the server and need access to pull the required Dell report.

In T251586#6114018, @Cmjohnson wrote:

@elukey I need to power this server off, I am not able to reach the mgmt/idrac for the server and need access to pull the required Dell report.

We can surely arrange this, let me know when it is a good time for you. I can access the mgmt serial though via cumin, do we need to power the server down?

@elukey I am not sure what was wrong yesterday but I didn't have any issues accessing the idrac today and downloaded the report. I opened the ticket with Dell.

Ticket #SR1024511445

the disk was replaced and is rebuilding...resolving the task, if it fails please let me know. The tracking number for the return disk is
USPS 9202 3946 5301 2445 77393 41
FEDEX 9611918 2393026 81239367

elukey@kafka-jumbo1001:~$ sudo megacli -LDPDInfo -aAll | grep State
State               : Optimal
Foreign State: None
Foreign State: None
State               : Optimal
Foreign State: None
Foreign State: None
Foreign State: None
Foreign State: None
Foreign State: None
Foreign State: None
Foreign State: None
Foreign State: None
Foreign State: None
Foreign State: None
Foreign State: None
Foreign State: None

All good!

Degraded RAID on kafka-jumbo1001Closed, ResolvedPublicActions

Description

Event Timeline

Degraded RAID on kafka-jumbo1001
Closed, ResolvedPublic
Actions