Page MenuHomePhabricator

CloudVPS: create wmcs-vm-fsck script
Closed, DeclinedPublic

Description

The process for fixing VM disks is very manual right now, but it can be automated.

Specially the steps described in here: https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#method_2:_qemu-nbd
Can be automated by:

  • installing required packages by puppet
  • creating a wmcs-vm-fsck script (name TBD) wrapping all these steps

This is related to incident: https://wikitech.wikimedia.org/wiki/Incident_documentation/20190213-cloudvps

Related Objects

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 14 2019, 12:51 PM
aborrero renamed this task from CloudVPS: crete wmcs-vm-fsck script to CloudVPS: create wmcs-vm-fsck script.Feb 14 2019, 12:51 PM
aborrero triaged this task as High priority.
aborrero moved this task from To Triage to Follow-up/Actionables on the Wikimedia-Incident board.
aborrero moved this task from Inbox to Important on the cloud-services-team (Kanban) board.

Also, (just to link thinks together) we have documentation on what we did, in part here: https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#Mounting_an_instance_disk

@GTirloni updated it with the module load command so it actually works. Apparently something like this had happened before.

GTirloni removed a subscriber: GTirloni.Mar 21 2019, 9:06 PM

We totally forgot about this.

Bstorm updated the task description. (Show Details)Jul 23 2019, 3:44 PM
marilerr closed this task as Declined.Aug 24 2019, 3:17 AM
JJMC89 reopened this task as Open.Aug 24 2019, 3:19 AM
marilerr closed this task as Declined.Aug 24 2019, 3:30 AM
JJMC89 reopened this task as Open.Aug 24 2019, 3:33 AM
Andrew closed this task as Declined.Wed, Sep 25, 3:17 PM

Brooke took a stab at this but writing the script turns out to be non-trivial; this happens infrequently and we have good docs now so we're going to try to avoid writing this.