Page MenuHomePhabricator

Investigate GitLab backup size change Jun 2026
Closed, ResolvedPublic

Description

In T431656 it was noticed the backup is finishing much faster than the usual one hour maintenance window.

Also the backup metrics show a decrease in backup time starting Jun 16th 14:20 UTC.

Additionally the size of the backup also decreased from 127G to 82G.

The major version upgrade happened Jun 17 T426164#12030376 (one day later of the size change)

So we should find out why the backup time decreased and if any data is missing in the backup.

Event Timeline

This was probably related to cleanup of the flaky-tests project, mentioned in T428903#12020643. This project had a 100GB LFS file checked in and it was removed by rewriting the git history.

I'll try to diff the current backup with an older one before Jun 16th.

Jelto triaged this task as Medium priority.Jul 13 2026, 3:29 PM
Jelto moved this task from Incoming to Work in Progress on the Collaboration-Services board.
Jelto added a subscriber: thcipriani.

I restored an old backup from bacular and there is a significant size difference in the LFS backup:

jelto@gitlab1004:~$ sudo tar -tvf /var/tmp/bacula-restores/srv/gitlab-backup/1781355868_2026_06_13_18.11.5_gitlab_backup.tar
-rw-r--r-- git/git         543 2026-06-13 13:58 backup_information.yml
drwxr-xr-x git/git           0 2026-06-13 13:04 db/
-rw------- git/git  1096931798 2026-06-13 13:06 db/database.sql.gz
drwx------ git/git           0 2026-06-13 13:11 repositories/
drwxr-xr-x git/git           0 2026-06-13 13:06 repositories/manifests/
drwxr-xr-x git/git           0 2026-06-13 13:11 repositories/backup_ids/
-rw-r--r-- git/git           0 2026-06-13 13:11 repositories/backup_ids/1781355868_2026_06_13_18.11.5
-rw------- git/git   658424651 2026-06-13 13:11 uploads.tar.gz
-rw------- git/git         329 2026-06-13 13:11 pages.tar.gz
-rw------- git/git 27125067326 2026-06-13 13:58 lfs.tar.gz
-rw------- git/git    86786252 2026-06-13 13:58 terraform_state.tar.gz
-rw------- git/git       19898 2026-06-13 13:58 ci_secure_files.tar.gz
-rw------- git/git         329 2026-06-13 13:58 external_diffs.tar.gz

jelto@gitlab1004:~$ sudo tar -tvf /srv/gitlab-backup/1783947873_2026_07_13_19.0.4_gitlab_backup.tar
-rw-r--r-- git/git         541 2026-07-13 13:17 backup_information.yml
drwxr-xr-x git/git           0 2026-07-13 13:04 db/
-rw------- git/git  1184617714 2026-07-13 13:06 db/database.sql.gz
drwx------ git/git           0 2026-07-13 13:11 repositories/
drwxr-xr-x git/git           0 2026-07-13 13:06 repositories/manifests/
drwxr-xr-x git/git           0 2026-07-13 13:11 repositories/backup_ids/
-rw-r--r-- git/git           0 2026-07-13 13:11 repositories/backup_ids/1783947873_2026_07_13_19.0.4
-rw------- git/git   659206578 2026-07-13 13:12 uploads.tar.gz
-rw------- git/git         329 2026-07-13 13:12 pages.tar.gz
-rw------- git/git  5262899057 2026-07-13 13:17 lfs.tar.gz
-rw------- git/git    87638835 2026-07-13 13:17 terraform_state.tar.gz
-rw------- git/git       19898 2026-07-13 13:17 ci_secure_files.tar.gz
-rw------- git/git         329 2026-07-13 13:17 external_diffs.tar.gz

lfs.tar.gz size went down from 27G to 5G. The other data is mostly unchanged.

So the cleanup of flaky-tests project (T428903#12020643) reduced the backup size and duration significant. The full backup size was 63G and went down to 42G, the backup duration went down from 1h to 15m.

Additionally the size of the backup also decreased from 127G to 82G.

That also makes sense because we store two GitLab backups in one bacula backup.

I'll resolve the task. cc @thcipriani