We know from T405224 that we can recover the entire cluster from the K3s data volumes. Additionally, the folks over at Cloud have told us in the past that it's not possible to automate/schedule the creation of OpenStack snapshots natively.

My proposal is:

• Increase disk quota capacity to allow us to create backup volumes for all data volumes
• Scheduled backups: Create systemd timer that rsyncs the data volumes every night from data volumes to backup volumes
• Recovery: Rsync back from backup volumes to data volumes

The timer will have as usual a separate service unit that does the actual work. This service can be one-shot'd to create a backup on demand prior to e.g. the K8s cluster upgrade we're planning

Looks like we'll need an extra 350G of volume space to do this + the tofu work to create the systemd timers. Let's double check volume space and request the change when we're ready.

Disk quotas for both catalyst and catalyst-dev will need to be raised by 320GB from the current 1200GB to a total of 1520GB

jnuche opened https://gitlab.wikimedia.org/repos/test-platform/catalyst/catalyst-tofu/-/merge_requests/36

Preliminary steps for the creation of K3s data volumes backups

jnuche merged https://gitlab.wikimedia.org/repos/test-platform/catalyst/catalyst-tofu/-/merge_requests/36

Preliminary steps for the creation of K3s data volumes backups

jnuche opened https://gitlab.wikimedia.org/repos/test-platform/catalyst/catalyst-tofu/-/merge_requests/37

back up K3s cluster data every night at 3:30 UTC

jnuche merged https://gitlab.wikimedia.org/repos/test-platform/catalyst/catalyst-tofu/-/merge_requests/37

back up K3s cluster data every night at 3:30 UTC