Page MenuHomePhabricator

documented procedure for replacing disks in software RAID servers
Closed, ResolvedPublic

Description

As part of T215183 let's figure out what needs to be done so that drive swaps in software RAID machines also involve reinstalling a bootloader on the new disk.

The 'easy' thing to do is to just add a note in the DCops runbook, but the section that one would add such a note to doesn't seem to exist yet.

I don't even know if it is usually dcops or the machine/service owner that repairs the MD device after a reinstall, though. (But in some glorious future it would be great to have some sort of automation there, which could also do this.)

Rob, thoughts?

Event Timeline

RobH added subscribers: Papaul, • Cmjohnson, wiki_willy.

I'm not quite sure what the other dc opsen do these days. Back when I was primary at a site, I tended to send the rebuild commands, and also the manual bootloader installation commands, but we had less servers and far far less SRE team members =]

@Cmjohnson & @Papaul: When you swap out a disk in a software raid, do you rebuild the raid or does that go back to the service owner to do?

(Edit Addition: I suspect it depends on who the service owners are, and if they can do it or if DC ops has to do on their behalf, but I'm not sure.)

Agreed automating this is ideal. Something where once a new disk is installed, a script can be triggered to re-sync the md devices and install a bootloader onto the new disk.

RobH triaged this task as Medium priority.Sep 9 2020, 12:39 AM

For me it goes back to the service owner. Also i think all the SW raid now has the bootloader installed all both disks if in case the server has 2 disks but not sure.

RobH closed this task as Resolved.EditedJul 1 2021, 6:49 PM

This is now documented on https://wikitech.wikimedia.org/wiki/SRE/Dc-operations/Sw_raid_rebuild_directions

The responsibility of it belonging to the service owners or to DC ops can be discussed here, but its not really something that I'd be in charge of, rather it would be a mgmt call. I'll add this to the discussion list of our meeting later today.

RobH removed RobH as the assignee of this task.
RobH claimed this task.