My only input is that I would delay starting this until all other fixes are in place, to save time and just test it once.

Change #1306166 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/puppet@production] backup: exclude lucene index from Gerrit backups

https://gerrit.wikimedia.org/r/1306166

Change #1306166 abandoned by Arnaudb:

[operations/puppet@production] backup: exclude lucene index from Gerrit backups

Reason:

https://phabricator.wikimedia.org/T430775

https://gerrit.wikimedia.org/r/1306166

The backup is being conducted while the service is actively writing to LFS object files (rarely happens), git packfiles and objects. We also have git garbage collection triggering on a weekly basis.

Thus potentially the backup could end up saving files that are being written to / are empty. Unless there is some locking mechanism by Gerrit jGit and bacula waits for the lock to be released somehow.

Maybe we would need a cold backup (aka with the service offline?)

We do have copies of git and LFS objects replicated to Gerrit replicas though and they should be identical (the files themselves are not, but the data should). Maybe the replicated data can be used as a source for offline backing up?

@hashar Thanks, but I think the hot-write concern is mostly covered.

Git never writes in place: loose objects, packfiles, refs and LFS objects are all written to a temp name and rename()d atomically. A hot backup can't capture a half-written file under its final name. Worst case is an orphan tmp_pack_*, or a .pack without its .idx, both of which git ignores. There is no lock to wait on and none needed. The GC collision also doesn't happen for the full backup. We run [gc] interval = 1 week, startTime = Sat 02:00 with [receive] autogc = false, while gerrit-repo-data uses Hourly-Tue-ReposEqiad, i.e. full on Tue 02:05 plus hourly incrementals. GC only ever overlaps incrementals.

What is real is the other thing: a multi-hour backup is crash-consistent, not point-in-time, and our jobdefaults leave accurate = no, so a restore replays the union of files in the cycle rather than the deletions. For git that is almost always harmless (duplicate loose objects plus a new packfile, which git does not care about). The one plausible loss: a loose ref captured at X, updated to Y and packed into packed-refs within the same hour. On the remedies: a cold backup means hours of downtime for our volume, so the answer there is an FS/LVM snapshot backed up out of band. Backing up from a replica doesn't fix consistency by itself (a replica writes continuously too) and replication is async and best-effort, so we'd be backing up a copy whose completeness we don't verify. It only helps if we can quiesce a dedicated replica, which is its own project.

Concretely: this is orthogonal to the fileset exclusions, since the same exposure existed before them, so I'd rather not widen the scope here. But the restore test is the cheapest way to get data on it, so I can add git fsck --full (or the jgit equivalent) on a sample of restored repos (All-Users plus a large one such as mediawiki/core) to the acceptance criteria.

I would agree with the snapshotting option- this can be implemented in pre-backup and post-backup bacula hooks to create and free the snapshots. Given incremental backups take minutes to run now, it shouldn't add a huge amount of disk resources and would ensure consistency, assuming lvm is already setup.

LVM is indeed setup, I'll file a task to track this effort so this one stays focused on the restore validation itself

Thanks for the answer!