Page MenuHomePhabricator

[etcd,infra] Find a backup solution for the etcd database
Open, HighPublic


Now that we start having more and more data stored in etcd, like envvars, builds, jobs and such, we might want to look into doing some kind of backup for the etcd database.

Docs on how to do a backup:

Some options mentioned on irc:

  • File backup with to a different server
    • Supports s3, so we might move to object storage when it's available
  • File backup with a new bacula setup in Toolforge
  • Manual scripts file backup to a different server
  • VM image snapshot to bacula (from the hypervisor)
    • Might need some syncing with the process that does the db dump

Add more if you have more ideas!

Event Timeline

dcaro triaged this task as High priority.Jun 20 2023, 3:23 PM
dcaro created this task.
dcaro added a project: Toolforge.

What if we backup the data to a cinder volume. I know etcd servers run in local storage hypervisors, but I'm not sure if that means they cannot use cinder?

If so, we could just:

  • mount a volume via puppet
  • have a timer to generate the snapshot and place it in the cinder volume
dcaro renamed this task from [etcd] Find a backup solution for the etcd database to [etcd,infra] Find a backup solution for the etcd database.Mar 18 2024, 11:01 AM