Page MenuHomePhabricator

CloudVPS: create wmcs-vm-fsck script
Open, HighPublic

Description

The process for fixing VM disks is very manual right now, but it can be automated.

Specially the steps described in here: https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/VM_images#method_2:_qemu-nbd
Can be automated by:

  • installing required packages by puppet
  • creating a wmcs-vm-fsck script (name TBD) wrapping all these steps

This is related to incident: https://wikitech.wikimedia.org/wiki/Incident_documentation/20190213-cloudvps

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 14 2019, 12:51 PM
aborrero triaged this task as High priority.Feb 14 2019, 12:51 PM
aborrero moved this task from To Triage to Follow-up/Actionables on the Wikimedia-Incident board.
aborrero moved this task from Inbox to Important on the cloud-services-team (Kanban) board.
aborrero renamed this task from CloudVPS: crete wmcs-vm-fsck script to CloudVPS: create wmcs-vm-fsck script.

Also, (just to link thinks together) we have documentation on what we did, in part here: https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#Mounting_an_instance_disk

@GTirloni updated it with the module load command so it actually works. Apparently something like this had happened before.

GTirloni removed a subscriber: GTirloni.Mar 21 2019, 9:06 PM