Page MenuHomePhabricator
Create Task
Maniphest T320482

Degraded RAID on elastic2052
Closed, ResolvedPublic2 Estimated Story Points
Actions

Description

TASK AUTO-GENERATED by Nagios/Icinga RAID event handler

A degraded RAID (md) was detected on host elastic2052. An automatic snapshot of the current RAID status is attached below.

Please sync with the service owner to find the appropriate time window before actually replacing any failed hardware.

CRITICAL: State: degraded, Active: 5, Working: 5, Failed: 1, Spare: 0

$ sudo /usr/local/lib/nagios/plugins/get-raid-status-md
Personalities : [raid0] [raid1] [linear] [multipath] [raid6] [raid5] [raid4] [raid10] 
md1 : active (auto-read-only) raid1 sdb3[1] sda3[0]
      999424 blocks super 1.2 [2/2] [UU]
      	resync=PENDING
      
md0 : active raid1 sda2[0](F) sdb2[1]
      78058496 blocks super 1.2 [2/1] [_U]
      
md2 : active raid0 sdb4[1] sda4[0]
      2966525952 blocks super 1.2 512k chunks
      
unused devices: <none>

Details

Other Assignee
RKemper

Related Objects

Event Timeline

ops-monitoring-bot created this task.Oct 11 2022, 9:49 AM
Restricted Application added subscribers: Marostegui, Aklapper. · View Herald TranscriptOct 11 2022, 9:49 AM
bking assigned this task to RKemper.Oct 11 2022, 4:46 PM
bking updated Other Assignee, added: bking.
bking added a project: Discovery-Search.
Stashbot added a comment.Oct 11 2022, 4:54 PM

Mentioned in SAL (#wikimedia-operations) [2022-10-11T16:54:38Z] <bking@cumin1001> START - Cookbook sre.hosts.downtime for 14 days, 0:00:00 on elastic2052.codfw.wmnet with reason: T320482

Stashbot added a comment.Oct 11 2022, 4:55 PM

Mentioned in SAL (#wikimedia-operations) [2022-10-11T16:55:03Z] <bking@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 14 days, 0:00:00 on elastic2052.codfw.wmnet with reason: T320482

bking added subscribers: Papaul, bking.Oct 11 2022, 4:55 PM

Looks like there is a documented procedure for DC Ops to follow.

@Papaul I've downtimed the host for the next 2 weeks, feel free to replace the disk at your convenience. If a reimage makes the procedure easier, feel free to do that as well.

bking removed RKemper as the assignee of this task.Oct 11 2022, 4:55 PM
bking updated Other Assignee, removed: bking.
bking added a subscriber: RKemper.
Papaul added a comment.Oct 11 2022, 8:06 PM

@bking this host is out of warranty. If it is a critical host you will have to let us know and request to purchase a disk. Another option is to check also if we have any disk similar from the decommissioned nodes that we can use.
Thanks.

Gehel edited projects, added Discovery-Search (Current work); removed Discovery-Search.Oct 12 2022, 1:49 PM
Gehel moved this task from Incoming to In Progress on the Discovery-Search (Current work) board.
MPhamWMF assigned this task to RKemper.Oct 17 2022, 3:21 PM
MPhamWMF updated Other Assignee, added: RKemper.
RKemper set the point value for this task to 2.Oct 24 2022, 3:29 PM
RKemper added a comment.Oct 25 2022, 7:28 PM
In T320482#8309495, @Papaul wrote:

@bking this host is out of warranty. If it is a critical host you will have to let us know and request to purchase a disk. Another option is to check also if we have any disk similar from the decommissioned nodes that we can use.
Thanks.

@Papaul We very recently decom'd these hosts: https://phabricator.wikimedia.org/T321243

Is one of their disks available to grab? If not, we could just decommission this host since I believe it only has about a year of lifespan left until refresh.

Papaul moved this task from Backlog to Hardware Failure / Troubleshoot on the ops-codfw board.Oct 30 2022, 6:59 PM

I will check when i am on site tomorrow.

Papaul added a comment.Oct 31 2022, 2:51 PM

@RKemper can you please double check this alert , looking at the disk here looks good to me no sign of failure led is green and showing activity.

jbond merged a task: T322042: Degraded RAID on elastic2052.Nov 4 2022, 11:30 AM
jbond added a subscriber: jbond.

FYI we are still reciving alerts for this disk to root

mdadm monitoring root@elastic2052.codfw.wmnet via wikimedia.org 
	
7:25 AM (5 hours ago)
	
to root
This is an automatically generated mail message from mdadm
running on elastic2052

A DegradedArray event had been detected on md device /dev/md/0.

Faithfully yours, etc.

P.S. The /proc/mdstat file currently contains the following:

Personalities : [raid1] [raid0] [linear] [multipath] [raid6] [raid5] [raid4] [raid10]
md0 : active raid1 sdb2[1]
      78058496 blocks super 1.2 [2/1] [_U]

md1 : active (auto-read-only) raid1 sda3[0] sdb3[1]
      999424 blocks super 1.2 [2/2] [UU]
        resync=PENDING

md2 : active raid0 sda4[0] sdb4[1]
      2966525952 blocks super 1.2 512k chunks

unused devices: <none>
RKemper added a comment.Nov 10 2022, 1:43 AM

@Papaul Yup per jbond's comment above we're still seeing the RAID issue. Could we try either rebuilding raid with the current disk, or swapping in a new one and rebuilding? (I suspect the latter is necessary but I'm not totally sure)

Gehel added a subscriber: Gehel.Nov 11 2022, 9:44 AM
In T320482#8385142, @RKemper wrote:

@Papaul Yup per jbond's comment above we're still seeing the RAID issue. Could we try either rebuilding raid with the current disk, or swapping in a new one and rebuilding? (I suspect the latter is necessary but I'm not totally sure)

Note that re-imaging this server from scratch is trivial. It might be easier than rebuilding the RAID array.

Papaul added a comment.Nov 14 2022, 6:17 PM

@RKemper @Gehel disk replaced

RKemper moved this task from In Progress to Needs Reporting on the Discovery-Search (Current work) board.Nov 17 2022, 7:54 PM

Reimaging now

ops-monitoring-bot added a comment.Nov 17 2022, 7:54 PM

Cookbook cookbooks.sre.hosts.reimage was started by ryankemper@cumin1001 for host elastic2052.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Nov 17 2022, 8:40 PM

Cookbook cookbooks.sre.hosts.reimage started by ryankemper@cumin1001 for host elastic2052.codfw.wmnet with OS bullseye completed:

  • elastic2052 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202211171954_ryankemper_1739297_elastic2052.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status failed -> active
Stashbot added a comment.Nov 21 2022, 2:48 PM

Mentioned in SAL (#wikimedia-operations) [2022-11-21T14:48:35Z] <gehel> repooling elastic2052 - T320482

Gehel closed this task as Resolved.Nov 21 2022, 2:48 PM
Content licensed under Creative Commons Attribution-ShareAlike 4.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL