Degraded RAID on db1046
Closed, ResolvedPublic
Actions

Assigned To

None

Authored By

	ops-monitoring-bot
	May 27 2017, 5:16 AM

Description

TASK AUTO-GENERATED by Nagios/Icinga RAID event handler

A degraded RAID (megacli) was detected on host db1046. An automatic snapshot of the current RAID status is attached below.

Please sync with the service owner to find the appropriate time window before actually replacing any failed hardware.

=== RaidStatus (does not include components in optimal state)
name: Adapter #0

	Virtual Drive: 0 (Target Id: 0)
	RAID Level: Primary-1, Secondary-0, RAID Level Qualifier-0
	State: =====> Degraded <=====
	Number Of Drives per span: 2
	Number of Spans: 6
	Current Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU

		Span: 1 - Number of PDs: 2

			PD: 1 Information
			Enclosure Device ID: 32
			Slot Number: 3
			Drive's position: DiskGroup: 0, Span: 1, Arm: 1
			Media Error Count: 91
			Other Error Count: 14
			Predictive Failure Count: =====> 182 <=====
			Last Predictive Failure Event Seq Number: 44085

				Raw Size: 279.396 GB [0x22ecb25c Sectors]
				Firmware state: =====> Failed <=====
				Media Type: Hard Disk Device
				Drive Temperature: 36C (96.80 F)

=== RaidStatus completed

Related Objects

Mentioned Here: T163778: Decommission db1022 (Was: db1022 broke while changing topology on s6- evaluate if to fix or directly decommission)
T164702: Decommission db1024
T166486: Decommission db1023
T166141: db1046 BBU looks faulty

Event Timeline

ops-monitoring-bot added projects: SRE, ops-eqiad.May 27 2017, 5:16 AM

ops-monitoring-bot subscribed.

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 27 2017, 5:16 AM

elukey triaged this task as High priority.May 27 2017, 9:04 AM

elukey added a project: Analytics-Kanban.

Volans added a project: DBA.May 29 2017, 11:14 AM

@elukey @Ottomata maybe this can be done along with: T166141

• Marostegui moved this task from Triage to In progress on the DBA board.May 30 2017, 5:57 AM

The disk was indeed bad...so it's been replaced. I don't know if I have enough bbu's to go around.. I am swapping them out of decom'd servers.

If it helps there are three more servers totally ready for you to decomm them: T166486 T163778 T164702

elukey@db1046:~$ sudo megacli -pdrbld -showprog -physdrv\[32:3\] -aALL

Rebuild Progress on Device at Enclosure 32, Slot 3 Completed 35% in 9 Minutes.

Exit Code: 0x00

• Cmjohnson moved this task from Backlog to High Priority Task on the ops-eqiad board.May 30 2017, 4:31 PM

This is now back to Optimal

root@db1046:~# megacli -LDPDInfo -aAll

Adapter #0

Number of Virtual Disks: 1
Virtual Drive: 0 (Target Id: 0)
Name                :
RAID Level          : Primary-1, Secondary-0, RAID Level Qualifier-0
Size                : 1.633 TB
Sector Size         : 512
Mirror Data         : 1.633 TB
State               : Optimal

Degraded RAID on db1046Closed, ResolvedPublicActions

Description

Related Objects

Event Timeline

Degraded RAID on db1046
Closed, ResolvedPublic
Actions