Page MenuHomePhabricator

Evaluate generic backup tooling for object storage buckets
Open, Needs TriagePublic

Description

In T378922: Migrate gitlab storage to apus (also: backups from S3?) there was a need for per-bucket backups. The current tooling (ceph, bacula and puppet) does not support easy backups for a single bucket.

To work around this, a new Puppet module ceph::client::sync_local was implemented, which clones a bucket to the regular file system where existing backup tools like bacula can operate. This solution works but is not ideal and introduces some overhead.

A more generic and proper tool for creating bucket-level backups would be useful in the long run. However currently only GitLab has the need to backup object storage buckets afaik. So maybe it makes sense to wait and collect requirements for other services as well.

Event Timeline

A more generic and proper tool for creating bucket-level backups

We do have a backup system for media using an s3 backend (media-backups) , but it was created specifically for mediawiki file backups, we do not backup swift. A generic method would need to be created, ideally not creating a new backend, just new middleware.