Maniphest T206965

Degraded RAID on dbstore1002
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	ops-monitoring-bot
	Oct 14 2018, 11:27 PM

Description

TASK AUTO-GENERATED by Nagios/Icinga RAID event handler

A degraded RAID (megacli) was detected on host dbstore1002. An automatic snapshot of the current RAID status is attached below.

Please sync with the service owner to find the appropriate time window before actually replacing any failed hardware.

CRITICAL: 1 failed LD(s) (Degraded)

$ sudo /usr/local/lib/nagios/plugins/get_raid_status_megacli
=== RaidStatus (does not include components in optimal state)
name: Adapter #0

	Virtual Drive: 0 (Target Id: 0)
	RAID Level: Primary-1, Secondary-0, RAID Level Qualifier-0
	State: =====> Degraded <=====
	Number Of Drives per span: 2
	Number of Spans: 6
	Current Cache Policy: WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU

		Span: 1 - Number of PDs: 2

			PD: 0 Information
			Enclosure Device ID: 32
			Slot Number: 2
			Drive's position: DiskGroup: 0, Span: 1, Arm: 0
			Media Error Count: 1
			Other Error Count: 17
			Predictive Failure Count: 0
			Last Predictive Failure Event Seq Number: 0

				Raw Size: 1.090 TB [0x8bba0cb0 Sectors]
				Firmware state: =====> Failed <=====
				Media Type: Hard Disk Device
				Drive Temperature: 28C (82.40 F)

=== RaidStatus completed

Related Objects

Mentioned In: T213670: dbstore1002 Mysql errors
T212908: dbstore1002 lagged on creating database

Event Timeline

ops-monitoring-bot added projects: SRE, ops-eqiad.Oct 14 2018, 11:27 PM

ops-monitoring-bot subscribed.

Restricted Application added subscribers: • Banyek, • Marostegui. · View Herald TranscriptOct 14 2018, 11:27 PM

• Marostegui added a project: Analytics.Oct 15 2018, 5:08 AM

• Marostegui added a subscriber: elukey.

• Cmjohnson moved this task from Backlog to Up next on the ops-eqiad board.Oct 15 2018, 4:49 PM

@Cmjohnson this server is OOW but the replacement will take time to arrive (still in procurement..) and this host is really important for the research users. Do we have a spare disk that we can swap?

• fdans moved this task from Incoming to Operational Excellence on the Analytics board.Oct 18 2018, 5:01 PM

@elukey dbstore1002 is out of warranty and has 1.2T disks. I don't have disks this size but can replace with a 2TB disk..

• Cmjohnson moved this task from Up next to High Priority Task on the ops-eqiad board.Oct 25 2018, 3:21 PM

Peachey88 mentioned this in T212908: dbstore1002 lagged on creating database.Jan 4 2019, 9:08 AM

• Marostegui mentioned this in T213670: dbstore1002 Mysql errors.Jan 14 2019, 7:57 AM

In T206965#4694827, @Cmjohnson wrote:

@elukey dbstore1002 is out of warranty and has 1.2T disks. I don't have disks this size but can replace with a 2TB disk..

Let's do it
Thanks

elukey assigned this task to • Cmjohnson.Jan 14 2019, 3:52 PM

nshahquinn-wmf added a project: Product-Analytics.Jan 16 2019, 9:34 PM

nshahquinn-wmf moved this task from Triage to Tracking on the Product-Analytics board.

RAID is back to optimal!
@Cmjohnson was this disk replaced then?

root@dbstore1002:~# megacli -LDInfo -lALL -aALL


Adapter 0 -- Virtual Drive Information:
Virtual Drive: 0 (Target Id: 0)
Name                :
RAID Level          : Primary-1, Secondary-0, RAID Level Qualifier-0
Size                : 6.545 TB
Sector Size         : 512
Mirror Data         : 6.545 TB
State               : Optimal
Strip Size          : 256 KB
Number Of Drives per span:2
Span Depth          : 6
Default Cache Policy: WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU
Current Cache Policy: WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU

• Phabricator_maintenance moved this task from Backlog to Acknowledged on the SRE board.Jan 26 2019, 10:31 PM

@Marostegui no, I reseated the disk but I see it's optimal. I am resolving the task. If it comes back we can open it again.

• Cmjohnson closed this task as Resolved.Jan 30 2019, 10:10 PM

Thanks!

Degraded RAID on dbstore1002Closed, ResolvedPublicActions

Description

Related Objects

Event Timeline

Degraded RAID on dbstore1002
Closed, ResolvedPublic
Actions