Page MenuHomePhabricator
Create Task
Maniphest T260941

Practice restoring ceph backups
Closed, ResolvedPublic
Actions

Description

I'd like all the wmcs SREs to have the experience of restoring a backed up ceph VM.

We can use ceph-restore-practice.testlabs.eqiad1.wikimedia.cloud as a test subject. Restoration docs are at https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Instance_backups#Restoring

You'll know that the VM is back up if you can

$ ssh ceph-restore-practice.testlabs.eqiad1.wikimedia.cloud cat /proofoflife.txt

Once Brooke and Arturo have confirmed a successful backup (and amended the docs as needed) this task can be closed.

Related Objects
Search...

Event Timeline

Andrew created this task.Aug 20 2020, 8:26 PM
Restricted Application removed a project: Patch-For-Review. · View Herald TranscriptAug 20 2020, 8:26 PM
Andrew updated the task description. (Show Details)Aug 20 2020, 8:27 PM
Andrew reassigned this task from Andrew to Bstorm.Aug 20 2020, 8:35 PM
Andrew added a subscriber: nskaggs.
Bstorm added a comment.Aug 21 2020, 9:42 PM
bstorm@ceph-restore-practice:~$ cat /proofoflife.txt
You did it!  This is the VM that you were trying to restore.

Ok! The process worked. I'll add a note about the snapshots since they have to be removed to delete a volume, apparently.

Bstorm reassigned this task from Bstorm to aborrero.Aug 21 2020, 9:47 PM

@aborrero 's turn

Stashbot added a comment.Sep 7 2020, 3:24 PM

Mentioned in SAL (#wikimedia-cloud) [2020-09-07T15:24:07Z] <arturo> practicing disk restoration on ceph-restore-practice (T260941)

aborrero closed this task as Resolved.Sep 7 2020, 3:36 PM

Before start:

aborrero@ceph-restore-practice:~$ cat /proofoflife.txt 
You did it!  This is the VM that you were trying to restore.
aborrero@ceph-restore-practice:~$ sudo rm /proofoflife.txt 
aborrero@ceph-restore-practice:~$ cat /proofoflife.txt 
cat: /proofoflife.txt: No such file or directory
aborrero@ceph-restore-practice:~$ sudo poweroff

following the restoration workflow:

root@cloudvirt1024:~# backy2 ls 5624cd3e-eeb8-421c-b8e4-7b1056a61550_disk
    INFO: [backy2.logging] $ /usr/bin/backy2 ls 5624cd3e-eeb8-421c-b8e4-7b1056a61550_disk
+---------------------+-------------------------------------------+---------------------+------+-------------+--------------------------------------+-------+-----------+----------------------------+---------------------+
|         date        | name                                      | snapshot_name       | size |  size_bytes |                 uid                  | valid | protected | tags                       |        expire       |
+---------------------+-------------------------------------------+---------------------+------+-------------+--------------------------------------+-------+-----------+----------------------------+---------------------+
[..]
| 2020-09-07 02:00:53 | 5624cd3e-eeb8-421c-b8e4-7b1056a61550_disk | 2020-09-07T02:00:50 | 5120 | 21474836480 | f5eec49e-f0ad-11ea-8511-b02628295df0 |   1   |     0     | b_daily                    | 2020-09-14 00:00:00 |
+---------------------+-------------------------------------------+---------------------+------+-------------+--------------------------------------+-------+-----------+----------------------------+---------------------+
    INFO: [backy2.logging] Backy complete.

root@cloudvirt1024:~# backy2 restore f5eec49e-f0ad-11ea-8511-b02628295df0 rbd://eqiad1-compute/5624cd3e-eeb8-421c-b8e4-7b1056a61550_disk
    INFO: [backy2.logging] $ /usr/bin/backy2 restore f5eec49e-f0ad-11ea-8511-b02628295df0 rbd://eqiad1-compute/5624cd3e-eeb8-421c-b8e4-7b1056a61550_disk
   ERROR: [backy2.logging] Image already exists: 5624cd3e-eeb8-421c-b8e4-7b1056a61550_disk
Error opening restore target.
root@cloudvirt1024:~# rbd rm eqiad1-compute/5624cd3e-eeb8-421c-b8e4-7b1056a61550_disk
2020-09-07 15:21:55.468 7f3858ff9700 -1 librbd::image::PreRemoveRequest: 0x55b576311dd0 check_image_snaps: image has snapshots - not removing
Removing image: 0% complete...failed.
rbd: image has snapshots - these must be deleted with 'rbd snap purge' before the image can be removed.
root@cloudvirt1024:~# rbd --pool eqiad1-compute snap purge 5624cd3e-eeb8-421c-b8e4-7b1056a61550_disk
Removing all snapshots: 100% complete...done.
root@cloudvirt1024:~# rbd --pool eqiad1-compute snap ls eqiad1-compute/5624cd3e-eeb8-421c-b8e4-7b1056a61550_disk
root@cloudvirt1024:~# rbd rm eqiad1-compute/5624cd3e-eeb8-421c-b8e4-7b1056a61550_disk
Removing image: 100% complete...done.
root@cloudvirt1024:~# backy2 restore f5eec49e-f0ad-11ea-8511-b02628295df0 rbd://eqiad1-compute/5624cd3e-eeb8-421c-b8e4-7b1056a61550_disk
    INFO: [backy2.logging] $ /usr/bin/backy2 restore f5eec49e-f0ad-11ea-8511-b02628295df0 rbd://eqiad1-compute/5624cd3e-eeb8-421c-b8e4-7b1056a61550_disk
[..]
    INFO: [backy2.logging] Backy complete.

After the restoration:

root@cloudcontrol1004:~# openstack server start 5624cd3e-eeb8-421c-b8e4-7b1056a61550
aborrero@ceph-restore-practice:~$ cat /proofoflife.txt 
You did it!  This is the VM that you were trying to restore.

This procedure works just fine. I guess is also a prominent candidate for automation (script) if ever detect we use this often.

Thanks @Andrew and @Bstorm the docs are really nice!

Andrew added a comment.Sep 7 2020, 4:56 PM

That's @aborrero! It's nice to know that backups still exist after I've ignored the system for a few weeks.

Content licensed under Creative Commons Attribution-ShareAlike 4.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL