Context
In T430775#12105890 @hashar raised the consistency of hot backups: the backup walks /srv/gerrit while Gerrit is writing. Analysis on the parent task narrowed the real exposure down to time skew across files in a multi-hour job (crash-consistent, not point-in-time, accurate = no), with one plausible loss scenario around refs being packed mid-cycle.
In T430775#12177927 @jcrespo endorsed snapshotting: create and free an LVM snapshot from Bacula pre and post job hooks. Incrementals take minutes now, so the snapshot lifetime and COW cost stay small. This task implements that.
A single snapshot of the srv LV covers /srv/gerrit/git and /srv/gerrit/data atomically, which gives every backup a true point-in-time view and closes the consistency question for good. This also feeds the tiering/snapshot follow-up left open from T411583: Gerrit backups are growing.
What is already in place
- The gerrit hosts (gerrit[1-2]00[1-4]) install with partman standard.cfg + raid1-2dev.cfg: RAID1, one VG (vg0), /srv as an LV, guided_size 80%, so about 20% of the VG is unallocated by design. Free extents for the snapshot COW should be there; confirm with vgs on each host in case LVs were extended after provisioning.
- Bacula pre-job hooks have in-house precedent: backup::mysqlset (predump method) and backup::openldapset ship ClientRunBeforeJob scripts, and backup::set / bacula::client::job already accept extras.
- The fileset is gerrit-repo-data (modules/profile/manifests/backup/filesets.pp), an allowlist of /srv/gerrit/git and /srv/gerrit/data, scheduled Hourly-Tue-ReposEqiad (full Tue 02:05, hourly incrementals) from modules/profile/manifests/gerrit.pp.
Proposed shape
- Pre-job (ClientRunBeforeJob): lvcreate -s on vg0/srv, mount the snapshot read-only at a stable path (e.g. /mnt/gerrit-backup-snap). Include a guard that removes any stale snapshot left by a previous failed job before creating the new one.
- Post-job (ClientRunAfterJob, and the failure variant so cleanup also runs on aborted jobs): unmount and lvremove.
- Point the gerrit-repo-data fileset at the snapshot mountpoint instead of /srv/gerrit. Restores are unaffected in practice since restore jobs choose their own where= target, but document the path change.
- Bacula fails the job if ClientRunBeforeJob fails, which is the behavior we want: no silent fallback to backing up the live tree.
Open questions
- Snapshot COW size: fixed size vs all free extents. It only has to absorb writes landing during the job, minutes for incrementals, longer for the Tuesday full.
- Monitoring: alert if a gerrit-backup snapshot exists outside a backup window, or if COW usage crosses a threshold (a full COW invalidates the snapshot and the running job with it).
- Whether the same pattern should be applied to the home set on the same hosts (probably not: /home lives on the root LV and is low-churn).
Acceptance criteria
- Backups run from a snapshot on all gerrit hosts; no job backs up the live /srv/gerrit tree anymore.
- A failed or interrupted job leaves no mounted snapshot behind.
- One documented restore from a snapshot-based backup (can piggyback on the T430775: Validate Gerrit backup recoverability after the fileset exclusions restore test).