The proof of concept in T259192 is looking pretty good; let's move ahead with making a real backup setup.
- identify hardware for this. 20Tb would be a nice amount of space to start with. Going to try to use cloudvirt1024. As a ceph node, its drives are idle; we'll see if running backup jobs interferes with VM performance.
- identify more projects and/or VM types to exclude from backups so we don't get overwhelmed when more VMs move to ceph
- puppetize the patch in https://github.com/wamdam/backy2/pull/72 (if upstream doesn't apply it and build a new package)
- switch it on, starting with 7 days of backups
- make sure we have disk space monitoring for the backup storage
- document restoration process, make sure WMCS staff has experience with restoration Documentation is here: https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Instance_backups#Future_concerns
- add additional cloudvirts to the cluster
- backup each VM in two places (T267195 ?)