The process for fixing VM disks is very manual right now, but it can be automated.
Specially the steps described in here: https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#method_2:_qemu-nbd
Can be automated by:
This is related to incident: https://wikitech.wikimedia.org/wiki/Incident_documentation/20190213-cloudvps
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Unknown Object (Task) | |||||
| Resolved | Andrew | T216218 Cloud VPS outage on cloudvirt1024 and cloudvirt1018 due to storage failure | |||
| Declined | None | T216132 CloudVPS: create wmcs-vm-fsck script |
Also, (just to link thinks together) we have documentation on what we did, in part here: https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#Mounting_an_instance_disk
@GTirloni updated it with the module load command so it actually works. Apparently something like this had happened before.
Brooke took a stab at this but writing the script turns out to be non-trivial; this happens infrequently and we have good docs now so we're going to try to avoid writing this.