Now that we start having more and more data stored in etcd, like envvars, builds, jobs and such, we might want to look into doing some kind of backup for the etcd database.

Docs on how to do a backup:

• https://kubernetes.io/docs/tasks/administer-cluster/configure-upgrade-etcd/#backing-up-an-etcd-cluster

Some options mentioned on irc:

• File backup with https://restic.net/ to a different server
  • Supports s3, so we might move to object storage when it's available
• File backup with a new bacula setup in Toolforge
• Manual scripts file backup to a different server
• VM image snapshot to bacula (from the hypervisor)
  • Might need some syncing with the process that does the db dump

Add more if you have more ideas!