Compact Gerrit H2 databases to reclaim disk
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	ABran-WMF
	Jul 1 2026, 7:41 AM

Project Tags

Referenced Files

None

Subscribers

Description

Context

@hashar explained the bloat in T411583#12074313. H2 does not auto-vacuum freed blocks. So the persistent-cache .h2.db files grow a lot. An 11G git_file_diff.h2.db compacts to about 40M. -Dh2.maxCompactTime=15000 only compacts on Gerrit shutdown, up to 15s. Gerrit 3.10.6 never compacts while running. This is the same on-disk H2 bloat that first filled the root partition, per T333143: Move Gerrit data out of root partition.

Scope note. The persistent caches are now excluded from Bacula, per T411583: Gerrit backups are growing. So compaction no longer changes the caches' backup size. Its benefit is host disk usage now. That is why T425667: Investigate Gerrit root disk usage and logging is a parent. It still helps the one H2 db that is still backed up, db/account_patch_reviews.h2.db.

Options

Immediate. Works on 3.10.6. Force a longer compaction with a stop/start dance, as @hashar described in T411583#12074321:
- raise -Dh2.maxCompactTime (for example to 180000) via Puppet or the unit
- stop Gerrit. It compacts up to the old 15s value.
- start Gerrit. It now has the new value.
- stop Gerrit. It compacts up to 180s.
- restore the value via Puppet.
- start Gerrit.
Permanent. The H2 v2 backend in Gerrit 3.12 vacuums routinely. This is blocked on the Gerrit 3.12 upgrade. The upgrade task is TBD. It was not found on Phabricator yet. Link it once filed.

Acceptance criteria

The .h2.db files are measurably smaller on disk after compaction.
Long-term, auto-vacuum is tracked via the Gerrit 3.12 upgrade.

T333143: Move Gerrit data out of root partition. Move data off root. The original "H2 fills the root partition".
Gerrit 3.12 changelog. New H2 v2 storage backend for persistent caches.

Related Objects
Search...

Status	Assigned	Task
		Restricted Task
Resolved	Dzahn	T411895 gerrit behind CDN
Resolved	ABran-WMF	T420523 ProbeDown - Service gerrit2003:443 has failed probes (http_gerrit_tls_ip6)
		Restricted Task
		Restricted Task
Open	None	T392448 Upgrade to Gerrit 3.12
Open	None	T407557 OpenSSH 10.1+ warns that Wikimedia SSH does not use post-quantum key exchange algorithm
Open	None	T407844 Gerrit ssh daemon does not offer post-quantum kex leading to a warning with OpenSSH 10
		Restricted Task
Open	None	T379714 Upgrade to Gerrit 3.11
Open	None	T392465 Switch Gerrit from Java 17 to Java 21
Open	None	T384595 Upgrade Collab hosts to Bookworm
Resolved	ABran-WMF	T392464 Upgrade Gerrit hosts from Bullseye to Bookworm
Open	None	T387831 Standardize failover procedures for Collab services
Resolved	None	T393239 ProbeDown
Resolved	ABran-WMF	T387833 Gerrit switchover process
Resolved	hashar	T406762 gerrit2003 is trying to backup incrementally 3.5 million files every hour, clogging backus and filling in available disk space
Resolved	hashar	T411583 Gerrit backups are growing
Resolved	Dzahn	T425667 Investigate Gerrit root disk usage and logging
Resolved	hashar	T430774 Compact Gerrit H2 databases to reclaim disk

Event Timeline

ABran-WMF created this task.Jul 1 2026, 7:41 AM

Restricted Application removed a project: Patch-For-Review. · View Herald TranscriptJul 1 2026, 7:41 AM

ABran-WMF removed ABran-WMF as the assignee of this task.Jul 1 2026, 7:41 AM

ABran-WMF triaged this task as Medium priority.

ABran-WMF added a parent task: T425667: Investigate Gerrit root disk usage and logging.

ABran-WMF moved this task from Incoming to Backlog on the Collaboration-Services board.

ABran-WMF mentioned this in T411583: Gerrit backups are growing.Jul 1 2026, 7:48 AM

ABran-WMF edited projects, added: Gerrit, Release-Engineering-Team; removed: Gerrit (Gerrit 3.12).Jul 1 2026, 7:52 AM

This happens automatically when Gerrit is being stopped (more exactly when the H2 DB Driver is about to disconnect from the db). We had multiple controlled restart of Gerrit over the last few days and the files got shrunk:

-rw-r--r-- 1 gerrit gerrit 762M Jul  9 14:50 git_file_diff.h2.db
-rw-r--r-- 1 gerrit gerrit 767M Jul  9 14:50 gerrit_file_diff.h2.db
-rw-r--r-- 1 gerrit gerrit 774M Jul  9 14:50 conflicts.h2.db
-rw-r--r-- 1 gerrit gerrit 785M Jul  9 14:48 diff_intraline.h2.db
-rw-r--r-- 1 gerrit gerrit 832M Jul  9 14:49 change_kind.h2.db
-rw-r--r-- 1 gerrit gerrit 845M Jul  9 14:50 mergeability.h2.db
-rw-r--r-- 1 gerrit gerrit 1.9G Jul  9 14:49 diff_summary.h2.db
-rw-r--r-- 1 gerrit gerrit 2.2G Jul  9 14:49 comment_context.h2.db

This task was forked from T411583#12074321 which is that those files were too large for the backup system due to changing frequently. They are no more backed up after the paths to backup have been corrected.

Compact Gerrit H2 databases to reclaim diskClosed, ResolvedPublicActions