With the "apus" multi-site S3 cluster ready (per T279621), we're going to migrate gitlab's storage to it as a first production user. I understand the plan is to detach one of the production replicas and move it first by way of testing.
This task is to track the migration work (and any other related work that becomes necessary).
**Todo:**
[x] Rename the //thanos_storage_enabled// Hiera flag
[ ] Talk to @jcrespo about the backup strategy. Are we better off backing up from apus, or using the existing /srv/gitlab-backups arrangement – either way the amount of space would seem to be roughly the same
[ ] Detach an instance for testing and revive the test instances. Use this to test moving data over and back, and any backup cron jobs that we decide on
[x] enable object storage on `gitlab1003`
[x] sync all packages and artifacts
[x] do some tests
[x] backup runtime: around 11 minutes and 30GB of size
[x] download old packages
[x] latency etc.
[x] test read-only credentials
[x] disable object storage again
[ ] Deployment plan
[] empty object storage again and remove test/replica files
[] enable object storage on the production host
[] wait until sync is done
[] verify download of artifacts/packages still works
[] enable object storage for replicas