Page MenuHomePhabricator
Create Task
Maniphest T370258

Degraded RAID on mw2432
Closed, InvalidPublic
Actions

Description

TASK AUTO-GENERATED by Nagios/Icinga RAID event handler

A degraded RAID (md) was detected on host mw2432. An automatic snapshot of the current RAID status is attached below.

Please sync with the service owner to find the appropriate time window before actually replacing any failed hardware.

CRITICAL: State: degraded, Active: 1, Working: 1, Failed: 0, Spare: 0

$ sudo /usr/local/lib/nagios/plugins/get-raid-status-md
Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5] [raid4] [raid10] 
md0 : active raid1 sda1[0]
      937030656 blocks super 1.2 [2/1] [U_]
      bitmap: 3/7 pages [12KB], 65536KB chunk

unused devices: <none>

Related Objects

Event Timeline

ops-monitoring-bot created this task.Jul 17 2024, 11:29 AM
Restricted Application added a project: DC-Ops. · View Herald TranscriptJul 17 2024, 11:29 AM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Jhancock.wm moved this task from Backlog to Hardware Failure / Troubleshoot on the ops-codfw board.Jul 17 2024, 12:35 PM
Jhancock.wm subscribed.Jul 17 2024, 2:26 PM

submitted a service request for dell to replace the drive. will update if they do.

I'm pretty certain that the drive in question is the second one (in drive bay 1) but would appreciate verification if possible. Thanks

SR194189251.

Clement_Goubert subscribed.EditedJul 17 2024, 2:33 PM

Hi @Jhancock.wm very sorry for the noise, this is me trying to automate turning the RAID controller to HBA mode, there are no actual issues with the disk. I didn't know it would create a ticket for you automatically.

I had downtimed the host, but apparently that wasn't enough to stop icinga from creating a ticket anyway.

Jhancock.wm added a comment.Jul 17 2024, 2:42 PM

all good! I can cancel the ticket.

Clement_Goubert added a comment.Jul 17 2024, 4:45 PM

Please do :) I'll leave the task open so it doesn't open a new one when I inevitably break it again. For the record, any raid issue you'll get for mw2432, mw2433, mw2438, mw2439 until T358489: mw2420-mw2451 do have unnecessary raid controllers (configured) is resolved is probably me. I will try and find out why downtiming the host didn't stop the autocreation of the ticket, but if I don't I'll watch for them being auto-opened and comment on them asap.

Papaul assigned this task to Clement_Goubert.Jul 22 2024, 1:39 PM
Papaul subscribed.

@Clement_Goubert hello we are going to assign this task to you and remove the dc-ops tag on it since you are doing some testing on this server. You can resolve it when done.
Thank you

Papaul removed projects: ops-codfw, DC-Ops.Jul 22 2024, 1:39 PM
Clement_Goubert closed this task as Invalid.Jul 22 2024, 1:41 PM

@Papaul Yep, sorry, it can be closed now. Cheers.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits