Page MenuHomePhabricator

wmcs-cold-migrate doesn't completely clean up local VMs
Closed, ResolvedPublic

Description

After a successful migration the wmcs-cold-migrate script deletes instance files from the old hypervisor and informs nova about the move. Virsh doesn't know about the change, though, so it's left with crumbs of the old VMs, e.g.:

root@cloudvirt1014:~# sudo virsh list --all  
 Id    Name                           State
----------------------------------------------------
 -     i-000013dc                     shut off
 -     i-000013de                     shut off
 -     i-00001402                     shut off
 -     i-0000143e                     shut off

Those can be removed via 'virsh undefine <id>'. It's not especially easy to patch wmcs-cold-migrate to do so, though, because 'virsh undefine' needs to be run as root and we only have keys set up to run things as 'nova'.

The leftover bits are entirely harmless (and seem to go away after a reboot) but it's still a bit messy.

Event Timeline

Change 518748 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] wmcs-cold-migrate: use 'virsh undefine' to cleanup old VMs

https://gerrit.wikimedia.org/r/518748

Change 518748 merged by Andrew Bogott:
[operations/puppet@production] wmcs-cold-migrate: use 'virsh undefine' to cleanup old VMs

https://gerrit.wikimedia.org/r/518748

Andrew assigned this task to JHedden.

Thanks Jason!