We run backups on the gitlab primary periodically, then rsync with a --delete flag to ensure that /srv/gitlab-backups doesn't grow too large.
The downside to this is that when we are creating a backup or running a restore on the remote hosts, these backups can be impacted.
We should change how the rsync command is scoped to avoid deleting backups that are in progress
Change #1015323 had a related patch set uploaded (by EoghanGaffney; author: EoghanGaffney):
[operations/puppet@production] [gitlab] Narrow scope of gitlab backup rsync commands
Change #1015323 merged by EoghanGaffney:
[operations/puppet@production] [gitlab] Narrow scope of gitlab backup rsync commands
I reverted this change in this CR since it was running into an issue where it couldn't find /srv/gitlab_backup/*_gitlab_backup.tar. This is because the rsync is run as a systemd unit with an ExecStart which doesn't wrap commands in a shell, so no globbing will work. Alternative solutions are either to wrap it in something like sh -c, or to wrap the whole thing in a shell script which gets executed by the systemd service.
Change #1017823 had a related patch set uploaded (by EoghanGaffney; author: EoghanGaffney):
[operations/puppet@production] [gitlab] Add shell script to replace rsync bare commands
Change #1017823 merged by EoghanGaffney:
[operations/puppet@production] [gitlab] Add shell script to replace rsync bare commands
Change #1018234 had a related patch set uploaded (by EoghanGaffney; author: EoghanGaffney):
[operations/puppet@production] gitlab: Switch rsync command in timer to run script
Change #1018234 merged by EoghanGaffney:
[operations/puppet@production] gitlab: Switch rsync command in timer to run script
Change #1024287 had a related patch set uploaded (by EoghanGaffney; author: EoghanGaffney):
[operations/puppet@production] gitlab: Fix rsync includes/excludes for data backup
Change #1024287 merged by EoghanGaffney:
[operations/puppet@production] gitlab: Fix rsync includes/excludes for data backup