Maniphest T196490

Degraded RAID on db1065
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	ops-monitoring-bot
	Jun 5 2018, 5:46 PM

Description

TASK AUTO-GENERATED by Nagios/Icinga RAID event handler

A degraded RAID (megacli) was detected on host db1065. An automatic snapshot of the current RAID status is attached below.

Please sync with the service owner to find the appropriate time window before actually replacing any failed hardware.

CRITICAL: 1 failed LD(s) (Degraded)



$ sudo /usr/local/lib/nagios/plugins/get_raid_status_megacli

=== RaidStatus (does not include components in optimal state)

name: Adapter #0



	Virtual Drive: 0 (Target Id: 0)

	RAID Level: Primary-1, Secondary-0, RAID Level Qualifier-0

	State: =====> Degraded <=====

	Number Of Drives per span: 2

	Number of Spans: 6

	Current Cache Policy: WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU



		Span: 0 - Number of PDs: 2



			PD: 1 Information

			Enclosure Device ID: 32

			Slot Number: 1

			Drive's position: DiskGroup: 0, Span: 0, Arm: 1

			Media Error Count: 3

			Other Error Count: 0

			Predictive Failure Count: 0

			Last Predictive Failure Event Seq Number: 0



				Raw Size: 558.911 GB [0x45dd2fb0 Sectors]

				Firmware state: =====> Rebuild <=====

				Media Type: Hard Disk Device

				Drive Temperature: 45C (113.00 F)



=== RaidStatus completed

Related Objects

Mentioned Here: T195444: db1065 storage crash

Event Timeline

ops-monitoring-bot added projects: SRE, ops-eqiad.Jun 5 2018, 5:46 PM

ops-monitoring-bot subscribed.

Restricted Application added subscribers: • Marostegui, Aklapper. · View Herald TranscriptJun 5 2018, 5:46 PM

This disk has been replaced by @Cmjohnson as we were coming from: T195444#4230827

• Marostegui triaged this task as Medium priority.Jun 5 2018, 5:53 PM

• Marostegui moved this task from Triage to In progress on the DBA board.

root@db1065:~# megacli -PDRbld -ShowProg -PhysDrv [32:1] -aALL

Rebuild Progress on Device at Enclosure 32, Slot 1 Completed 56% in 31 Minutes.

• Cmjohnson moved this task from Backlog to High Priority Task on the ops-eqiad board.Jun 5 2018, 6:16 PM

All good!

Number of Virtual Disks: 1
Virtual Drive: 0 (Target Id: 0)
Name                :
RAID Level          : Primary-1, Secondary-0, RAID Level Qualifier-0
Size                : 3.271 TB
Sector Size         : 512
Mirror Data         : 3.271 TB
State               : Optimal
Strip Size          : 256 KB
Number Of Drives per span:2
Span Depth          : 6
Default Cache Policy: WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU
Current Cache Policy: WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU

• Vvjjkkii renamed this task from Degraded RAID on db1065 to 9kbaaaaaaa.Jul 1 2018, 1:06 AM

• Vvjjkkii reopened this task as Open.

• Vvjjkkii removed • Cmjohnson as the assignee of this task.

• Vvjjkkii raised the priority of this task from Medium to High.

• Vvjjkkii added projects: CheckUser, Connected-Open-Heritage-Batch-uploads (RAÄ-KMB_1_2017-02), Tamil-Sites, Gamepress, Hashtags, Jade, KartoEditor, Language-2018-Apr-June, New-Editor-Experiences, Mail, TCB-Team (now WMDE-TechWish).

• Vvjjkkii updated the task description. (Show Details)

• Vvjjkkii removed a subscriber: Aklapper.

• Marostegui renamed this task from 9kbaaaaaaa to Degraded RAID on db1065.Jul 2 2018, 5:03 AM

• Marostegui closed this task as Resolved.

• Marostegui assigned this task to • Cmjohnson.

• Marostegui lowered the priority of this task from High to Medium.

• Marostegui removed projects: TCB-Team (now WMDE-TechWish), Mail, New-Editor-Experiences, Language-2018-Apr-June, KartoEditor, Jade, Hashtags, Gamepress, Tamil-Sites, Connected-Open-Heritage-Batch-uploads (RAÄ-KMB_1_2017-02), CheckUser.

• Marostegui updated the task description. (Show Details)

Degraded RAID on db1065Closed, ResolvedPublicActions

Description

Related Objects

Event Timeline

Degraded RAID on db1065
Closed, ResolvedPublic
Actions