Page MenuHomePhabricator

Create backup service for glance images
Closed, ResolvedPublic

Description

This should be very similar to the backy2 job for VMs, only simpler.

This might also be an opportunity to experiment with benji

Event Timeline

Andrew created this task.Oct 18 2020, 9:55 PM
Restricted Application removed a project: Patch-For-Review. · View Herald TranscriptOct 18 2020, 9:55 PM

Change 634997 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] wmcs: add backup jobs for glance images on cloudcontrols

https://gerrit.wikimedia.org/r/634997

Change 634997 merged by Andrew Bogott:
[operations/puppet@production] wmcs: add backup jobs for glance images on cloudcontrols

https://gerrit.wikimedia.org/r/634997

Change 635004 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] wmcs: glance image backup bugfixes

https://gerrit.wikimedia.org/r/635004

Change 635004 merged by Andrew Bogott:
[operations/puppet@production] wmcs: glance image backup bugfixes

https://gerrit.wikimedia.org/r/635004

Change 635312 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] wmcs-backup-images.py: use admin rather than observer credentials

https://gerrit.wikimedia.org/r/635312

Change 635312 merged by Andrew Bogott:
[operations/puppet@production] wmcs-backup-images.py: use admin rather than observer credentials

https://gerrit.wikimedia.org/r/635312

Andrew triaged this task as Medium priority.Oct 20 2020, 4:19 PM
Andrew closed this task as Resolved.Oct 21 2020, 2:34 AM

I've just confirmed that backups of new images are created, and I've documented the restore process here:

https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Instance_backups#Restoring_a_lost_Glance_image

Right now we back up each image in three places (on each cloudcontrol node) and store a 7-day history. The history is a bit silly since images don't change after creation, but since the backups are incremental the disk usage will be minimal. Unless the too-frequent backups turn out to be causing trouble with Ceph I'd say this is done.

Change 635676 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] wmcs instance and image backups: move some jobs one hour later

https://gerrit.wikimedia.org/r/635676

Change 635676 merged by Andrew Bogott:
[operations/puppet@production] wmcs instance and image backups: move some jobs one hour later

https://gerrit.wikimedia.org/r/635676