Page MenuHomePhabricator
Create Task
Maniphest T409638

Degraded RAID on db1171
Closed, ResolvedPublic
Actions

Description

TASK AUTO-GENERATED by Nagios/Icinga RAID event handler

A degraded RAID (megacli) was detected on host db1171. An automatic snapshot of the current RAID status is attached below.

Please sync with the service owner to find the appropriate time window before actually replacing any failed hardware.

CRITICAL: 1 failed LD(s) (Degraded)

$ sudo /usr/local/lib/nagios/plugins/get-raid-status-megacli
=== RaidStatus (does not include components in optimal state)
name: Adapter #0

	Virtual Drive: 0 (Target Id: 0)
	RAID Level: Primary-1, Secondary-0, RAID Level Qualifier-0
	State: =====> Degraded <=====
	Number Of Drives: 10
	Number of Spans: 1
	Current Cache Policy: WriteBack, ReadAhead, Direct, No Write Cache if Bad BBU

=== RaidStatus completed

Event Timeline

ops-monitoring-bot created this task.Nov 8 2025, 5:26 PM
Restricted Application added projects: DC-Ops, DBA. · View Herald TranscriptNov 8 2025, 5:26 PM
• Marostegui edited projects, added Data-Persistence, Data-Persistence-Backup; removed DBA.Nov 8 2025, 5:36 PM
• Marostegui added subscribers: jcrespo, • Marostegui.

@jcrespo this is a backup source. I would assume you want the disk replaced with a spare, but just tagging you to confirm.

Restricted Application added a project: DBA. · View Herald TranscriptNov 8 2025, 5:36 PM
jcrespo added a comment.Nov 10 2025, 9:15 AM

Yes, please, dc ops, file a servicing request or help us with a spare here.

Jclark-ctr subscribed.Mon, Nov 10, 11:38 AM

@jcrespo can this be swapped at anytime or do we need to schedule?

jcrespo added a comment.EditedMon, Nov 10, 11:40 AM

Go ahead if it doesn't require shutdown. If it requires or it is preferred, just let me know and I will perform it myself right now, will tell you when stopped. Otherwise it can be done at any time.

Jclark-ctr claimed this task.Mon, Nov 10, 1:17 PM
Jclark-ctr moved this task from Backlog to Hardware Failure / Troubleshoot on the ops-eqiad board.
Jclark-ctr added a comment.Mon, Nov 10, 2:15 PM

@jcrespo This server is out of warranty. I replaced the disk with one from a decommissioned server; the drive was erased prior to installation. No new errors are currently listed, aside from previous ones on the controller from 11/8. The RAID appears healthy, but iDRAC still shows a warning. There are no failed drives, and the rebuild is currently in progress. I will check it again later. Additionally, I updated the iDRAC firmware from version 4.40.00.00 to 7.00.00.182.

jcrespo added a comment.Mon, Nov 10, 3:05 PM

I will keep an eye on it until it gets rebuilt, thanks for the quick help. I will also have a look at the warnings.

jcrespo added a comment.Mon, Nov 10, 3:27 PM

I saw the warnings, but I see no problem on the logs, other than it detecting your disk change and firmware update. Once the disk rebuild finishes I will do a server restart to see if that clears the warnings.

Jclark-ctr closed this task as Resolved.Mon, Nov 10, 9:29 PM

Idrac is showing SYSTEM IS HEALTHY after rebuilding.

ops-monitoring-bot added a comment.Wed, Nov 12, 9:21 AM

Icinga downtime and Alertmanager silence (ID=efc351b5-c9f2-4bac-808a-7ec10adef598) set by jynus@cumin1003 for 2:00:00 on 1 host(s) and their services with reason: Restart

db1171.eqiad.wmnet
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits