Degraded RAID on cloudvirt1018
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	ops-monitoring-bot
	Nov 28 2021, 2:32 PM

Description

TASK AUTO-GENERATED by Nagios/Icinga RAID event handler

A degraded RAID (megacli) was detected on host cloudvirt1018. An automatic snapshot of the current RAID status is attached below.

Please sync with the service owner to find the appropriate time window before actually replacing any failed hardware.

CRITICAL: 1 failed LD(s) (Degraded)

$ sudo /usr/local/lib/nagios/plugins/get-raid-status-megacli
=== RaidStatus (does not include components in optimal state)
name: Adapter #0

	Virtual Drive: 0 (Target Id: 0)
	RAID Level: Primary-1, Secondary-0, RAID Level Qualifier-0
	State: =====> Degraded <=====
	Number Of Drives: 8
	Number of Spans: 1
	Current Cache Policy: WriteBack, ReadAhead, Direct, No Write Cache if Bad BBU

		Span: 0 - Number of PDs: 8

			PD: 1 Information
			Enclosure Device ID: 32
			Slot Number: 8
			Drive's position: DiskGroup: 0, Span: 0, Arm: 1
			Media Error Count: 0
			Other Error Count: 0
			Predictive Failure Count: 0
			Last Predictive Failure Event Seq Number: 0

				Raw Size: 1.455 TB [0xba4d4ab0 Sectors]
				Firmware state: =====> Rebuild <=====
				Media Type: Solid State Device
				Drive Temperature: 26C (78.80 F)

=== RaidStatus completed

Related Objects

Mentioned Here: T296790: decommission cloudvirt1018.eqiad.wmnet
T296664: Our need for cloudvirt hypervisors with local disks

Event Timeline

ops-monitoring-bot created this task.Nov 28 2021, 2:32 PM

This is one of the localdisk hypervisors we use for Toolforge/Toolsbeta etcd, thankfully not a ToolsDB server

Restricted Application edited projects, added cloud-services-team (Kanban); removed cloud-services-team. · View Herald TranscriptNov 28 2021, 2:43 PM

Mentioned in SAL (#wikimedia-cloud) [2021-11-28T17:48:48Z] <andrewbogott> moved cloudvirt1018 out of the 'localstorage' aggregate and into 'maintenance' for T296592. It will need to be moved back after the raid is rebuilt.

wiki_willy assigned this task to Cmjohnson.Nov 29 2021, 7:43 AM

@wiki_willy @RobH this server is out of warranty, they have a 1.6TB SSD that has failed. I recommend buying a new SSD.

Cmjohnson moved this task from Backlog to Hardware Failure / Troubleshoot on the ops-eqiad board.Nov 29 2021, 3:35 PM

Cmjohnson moved this task from Hardware Failure / Troubleshoot to Lower Priority Items on the ops-eqiad board.

This server is 5 years old in May 2022, do we want to throw new hardware in a host going away in less than 6 months or just move the refresh to Q3?

++ @nskaggs & @Andrew - since this server is scheduled for a refresh in Q4 (line 130 on the procurement doc), are you guys ok with not fixing this, and just sticking with a Q4 (or Q3) refresh schedule? Thanks, Willy

@wiki_willy we can probably live without it for a few months; I'll do a bit of research to figure out what would be a good replacement (it was one in a set of three handling a weird use-case)

Andrew claimed this task.Nov 29 2021, 5:47 PM

@wiki_willy Is their an advantage to getting the normal refresh order in now? Would you recommend it? I know prices and lead times have been an issue this year.

Hi @nskaggs - sure, we can totally order the "refresh of cloudvirt101[6-8]" now, and have it to arrive in early Q3 if you want. The lead times with vendors have been fluctuating a bit, but I can definitely work with Finance to bump the forecast on this one a bit earlier on the budget. Does that work for you?

In T296592#7534843, @nskaggs wrote:

@wiki_willy Is their an advantage to getting the normal refresh order in now? Would you recommend it? I know prices and lead times have been an issue this year.

I just did some digging and some thinking and I don't see a reason why we need to rush this refresh. Most likely these hosts will be replaced with thinvirts which won't meet the 'localdisk' role that this host was playing anyway.

I've opened T296664 to discuss the future of fatvirts but we need to resolve that separately from the cloudvirt1018 refresh.

Thanks @Andrew, sounds good. I'll leave it on the schedule for a Q4 refresh for now, but let me know if you end up needing it earlier, and we can always adjust and push things up then. Thanks, Willy

In T296592#7535085, @Andrew wrote:

I just did some digging and some thinking and I don't see a reason why we need to rush this refresh. Most likely these hosts will be replaced with thinvirts which won't meet the 'localdisk' role that this host was playing anyway.

I've opened T296664 to discuss the future of fatvirts but we need to resolve that separately from the cloudvirt1018 refresh.

@Andrew If we're going to leave this be, is it okay to close this task? Your server refresh can be tracked in a separate task

RobH removed a subscriber: RobH.Nov 30 2021, 8:53 PM

Are we going to simply decommission this machine then and remove it from the rack? If so, then I would consider that the resolution of this ticket. But I think we should decide what the outcome of the existing server will be before closing.

Yep, let's just decom. I'll open a ticket for that and then close this

https://phabricator.wikimedia.org/T296790

Degraded RAID on cloudvirt1018Closed, ResolvedPublicActions

Description

Related Objects

Event Timeline

Degraded RAID on cloudvirt1018
Closed, ResolvedPublic
Actions