Page MenuHomePhabricator
Create Task
Maniphest T433030

Degraded RAID on an-presto1013
Closed, ResolvedPublic
Actions

Description

TASK AUTO-GENERATED by Nagios/Icinga RAID event handler

A degraded RAID (broadcom) was detected on host an-presto1013. An automatic snapshot of the current RAID status is attached below.

Please sync with the service owner to find the appropriate time window before actually replacing any failed hardware.

communication: 0 OK : controller: 1 Needs Attention : physical_disk: 1 Failed : virtual_disk: 1 Dgrd : bbu: 0 OK : enclosure: 0 OK : CLI Version = 007.1910.0000.0000 Oct 08, 2021

$ sudo /usr/local/lib/nagios/plugins/get-raid-status-broadcom
Failed to execute '['/usr/lib/nagios/plugins/check_nrpe', '-4', '-H', 'an-presto1013', '-c', 'get_raid_status_broadcom']': RETCODE: 2
STDOUT:
communication: 0 OK ; controller: 1 Needs Attention ; physical_disk: 1 Failed ; virtual_disk: 1 Dgrd ; bbu: 0 OK ; enclosure: 0 OK ; CLI Version = 007.1910.0000.0000 Oct 08, 2021
Operating system = Linux 5.10.0-43-amd64
Controller = 0
Status = Success
Description = Show Drive Group Succeeded


TOPOLOGY :
========

-------------------------------------------------------------------------------
DG Arr Row EID:Slot DID Type   State  BT       Size PDC  PI SED DS3  FSpace TR 
-------------------------------------------------------------------------------
 0 -   -   -        -   RAID10 Dgrd   N   21.829 TB dflt N  N   dflt N      N  
 0 0   -   -        -   RAID1  Dgrd   N   21.829 TB dflt N  N   dflt N      N  
 0 0   0   64:0     5   DRIVE  Failed N    3.637 TB dflt N  N   dflt -      N  
 0 0   1   64:1     0   DRIVE  Onln   N    3.637 TB dflt N  N   dflt -      N  
 0 0   2   64:2     1   DRIVE  Onln   N    3.637 TB dflt N  N   dflt -      N  
 0 0   3   64:3     3   DRIVE  Onln   N    3.637 TB dflt N  N   dflt -      N  
 0 0   4   64:4     14  DRIVE  Onln   N    3.637 TB dflt N  N   dflt -      N  
 0 0   5   64:5     8   DRIVE  Onln   N    3.637 TB dflt N  N   dflt -      N  
 0 0   6   64:6     4   DRIVE  Onln   N    3.637 TB dflt N  N   dflt -      N  
 0 0   7   64:7     10  DRIVE  Onln   N    3.637 TB dflt N  N   dflt -      N  
 0 0   8   64:8     2   DRIVE  Onln   N    3.637 TB dflt N  N   dflt -      N  
 0 0   9   64:9     6   DRIVE  Onln   N    3.637 TB dflt N  N   dflt -      N  
 0 0   10  64:10    7   DRIVE  Onln   N    3.637 TB dflt N  N   dflt -      N  
 0 0   11  64:11    11  DRIVE  Onln   N    3.637 TB dflt N  N   dflt -      N  
 1 -   -   -        -   RAID1  Optl   N  446.625 GB dflt N  N   dflt N      N  
 1 0   -   -        -   RAID1  Optl   N  446.625 GB dflt N  N   dflt N      N  
 1 0   0   64:12    13  DRIVE  Onln   N  446.625 GB dflt N  N   dflt -      N  
 1 0   1   64:13    12  DRIVE  Onln   N  446.625 GB dflt N  N   dflt -      N  
-------------------------------------------------------------------------------

DG=Disk Group Index|Arr=Array Index|Row=Row Index|EID=Enclosure Device ID
DID=Device ID|Type=Drive Type|Onln=Online|Rbld=Rebuild|Optl=Optimal|Dgrd=Degraded
Pdgd=Partially degraded|Offln=Offline|BT=Background Task Active
PDC=PD Cache|PI=Protection Info|SED=Self Encrypting Drive|Frgn=Foreign
DS3=Dimmer Switch 3|dflt=Default|Msng=Missing|FSpace=Free Space Present
TR=Transport Ready






STDERR:
None

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedVRiley-WMFT433030 Degraded RAID on an-presto1013
Unknown Object (Task)

Event Timeline

ops-monitoring-bot created this task.Jul 24 2026, 12:11 AM
Restricted Application added a project: DC-Ops. · View Herald TranscriptJul 24 2026, 12:11 AM
VRiley-WMF moved this task from Backlog to Hardware Failure / Troubleshoot on the ops-eqiad board.Jul 24 2026, 4:45 PM
VRiley-WMF added subscribers: BTullis, wiki_willy, RobH, VRiley-WMF.Jul 24 2026, 5:57 PM

@BTullis it looks like this disk has failed that was predicted in ticket https://phabricator.wikimedia.org/T408065

As previously mentioned, @RobH and @wiki_willy, we would need to order a replacment, as this unit doesn't have a warrenty and we have no 4TB SAS on spare.

VRiley-WMF claimed this task.Jul 24 2026, 5:58 PM
VRiley-WMF added a project: Data-Platform-SRE (2026-07-03 - 2026-07-31).
RobH mentioned this in Unknown Object (Task).Jul 30 2026, 5:32 PM
RobH added a subtask: Unknown Object (Task).
VRiley-WMF added a comment.Aug 5 2026, 5:33 PM

Currently waiting on HDD to arrive

VRiley-WMF moved this task from Hardware Failure / Troubleshoot to Blocked on the ops-eqiad board.Aug 5 2026, 5:33 PM
VRiley-WMF closed subtask Unknown Object (Task) as Resolved.Mon, Aug 10, 3:18 PM
Gehel edited projects, added: Data-Platform-SRE (2026-08-07 - 2026-08-28); removed: Data-Platform-SRE (2026-07-03 - 2026-07-31).Tue, Aug 11, 6:52 AM
VRiley-WMF changed the task status from Open to In Progress.Tue, Aug 11, 5:31 PM

Referring to https://phabricator.wikimedia.org/T408065 that these drives can be replaced without asking. Will replace the drive now.

VRiley-WMF closed this task as Resolved.Tue, Aug 11, 5:49 PM

Drive has been replaced and currently is rebuilding.

RobH reopened this task as Open.Tue, Aug 11, 6:02 PM

The raid is still in degraded status, we should keep this task open until its successful.

Related: failed drive on an-presto1007 from same order batch on T434505, as long as this raid successfully rebuilds we can order more of the same drive.

robh@an-presto1013:~$ sudo /usr/local/lib/nagios/plugins/get-raid-status-broadcom
communication: 0 OK ; controller: 1 Needs Attention ; physical_disk: 1 Rbld ; virtual_disk: 1 Dgrd ; bbu: 0 OK ; enclosure: 0 OK ; CLI Version = 007.1910.0000.0000 Oct 08, 2021
Operating system = Linux 5.10.0-43-amd64
Controller = 0
Status = Success
Description = Show Drive Group Succeeded


TOPOLOGY :
========

------------------------------------------------------------------------------
DG Arr Row EID:Slot DID Type   State BT       Size PDC  PI SED DS3  FSpace TR 
------------------------------------------------------------------------------
 0 -   -   -        -   RAID10 Dgrd  N   21.829 TB dflt N  N   dflt N      N  
 0 0   -   -        -   RAID1  Dgrd  N   21.829 TB dflt N  N   dflt N      N  
 0 0   0   64:0     15  DRIVE  Rbld  Y    3.637 TB dflt N  N   dflt -      N  
 0 0   1   64:1     0   DRIVE  Onln  N    3.637 TB dflt N  N   dflt -      N  
 0 0   2   64:2     1   DRIVE  Onln  N    3.637 TB dflt N  N   dflt -      N  
 0 0   3   64:3     3   DRIVE  Onln  N    3.637 TB dflt N  N   dflt -      N  
 0 0   4   64:4     14  DRIVE  Onln  N    3.637 TB dflt N  N   dflt -      N  
 0 0   5   64:5     8   DRIVE  Onln  N    3.637 TB dflt N  N   dflt -      N  
 0 0   6   64:6     4   DRIVE  Onln  N    3.637 TB dflt N  N   dflt -      N  
 0 0   7   64:7     10  DRIVE  Onln  N    3.637 TB dflt N  N   dflt -      N  
 0 0   8   64:8     2   DRIVE  Onln  N    3.637 TB dflt N  N   dflt -      N  
 0 0   9   64:9     6   DRIVE  Onln  N    3.637 TB dflt N  N   dflt -      N  
 0 0   10  64:10    7   DRIVE  Onln  N    3.637 TB dflt N  N   dflt -      N  
 0 0   11  64:11    11  DRIVE  Onln  N    3.637 TB dflt N  N   dflt -      N  
 1 -   -   -        -   RAID1  Optl  N  446.625 GB dflt N  N   dflt N      N  
 1 0   -   -        -   RAID1  Optl  N  446.625 GB dflt N  N   dflt N      N  
 1 0   0   64:12    13  DRIVE  Onln  N  446.625 GB dflt N  N   dflt -      N  
 1 0   1   64:13    12  DRIVE  Onln  N  446.625 GB dflt N  N   dflt -      N  
------------------------------------------------------------------------------

DG=Disk Group Index|Arr=Array Index|Row=Row Index|EID=Enclosure Device ID
DID=Device ID|Type=Drive Type|Onln=Online|Rbld=Rebuild|Optl=Optimal|Dgrd=Degraded
Pdgd=Partially degraded|Offln=Offline|BT=Background Task Active
PDC=PD Cache|PI=Protection Info|SED=Self Encrypting Drive|Frgn=Foreign
DS3=Dimmer Switch 3|dflt=Default|Msng=Missing|FSpace=Free Space Present
TR=Transport Ready
RobH added a comment.Tue, Aug 11, 6:03 PM

Actually, I'm torn on this.

R

aw Size: 3.638 TB [0x1d1c0beb0 Sectors]
Non Coerced Size: 3.637 TB [0x1d1b0beb0 Sectors]
Coerced Size: 3.637 TB [0x1d1b00000 Sectors]
Sector Size: 512
Firmware state: Rebuild
Device Firmware Level: EG03
Shield Counter: 0
Successful diagnostics completion on : N/A
SAS Address(0): 0x50000399287bdcd6
SAS Address(1): 0x0
Connected Port Number: 8(path0)
Inquiry Data: TOSHIBA MG04SCA40ENY EG0319X0A0K4F74E
FDE Capable: Not Capable
FDE Enable: Disable
Secured: Unsecured
Locked: Unlocked
Needs EKM Attention: No
Foreign State: None
Device Speed: 12.0Gb/s
Link Speed: 12.0Gb/s
Media Type: Hard Disk Device
Drive Temperature :25C (77.00 F)
PI Eligibility: No
Drive is formatted for PI information: Yes
PI: PI with type 2
Port-0 :
Port status: Active
Port's Linkspeed: 12.0Gb/s
Port-1 :
Port status: Active
Port's Linkspeed: 12.0Gb/s
Drive has flagged a S.M.A.R.T alert : No

It is rebuilding, we likely should keep the task open until the icinga alert clears.

RobH mentioned this in T434505: Degraded RAID on an-presto1007.Tue, Aug 11, 6:05 PM
VRiley-WMF closed this task as Resolved.Wed, Aug 12, 12:52 PM

The drive seems to have been fully rebuilt and is now in a healthy state

Gehel moved this task from Backlog - project to Done on the Data-Platform-SRE (2026-08-07 - 2026-08-28) board.Fri, Aug 14, 5:42 AM
Gehel moved this task from Done to Reported on the Data-Platform-SRE (2026-08-07 - 2026-08-28) board.Fri, Aug 14, 5:43 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits