Page MenuHomePhabricator
Create Task
Maniphest T411583

Gerrit backups are growing
Closed, ResolvedPublic
Actions

Description

Following up on T406762: gerrit2003 is trying to backup incrementally 3.5 million files every hour, clogging backus and filling in available disk space

The backup size increase is probably a side effect of the local-backup cookbook which duplicates critical git data locally to ensure a local stable state snapshot is quickly available. The most obvious fix for this could be to exclude the backup directory from bacula.

Related Objects
Search...

Event Timeline

ABran-WMF created this task.Dec 3 2025, 9:22 AM
jcrespo added a comment.Dec 3 2025, 10:25 AM

Question: I don't know the details of how are things are organized for resilience, but wouldn't the other host should have those files (and thus, backup would frequently have the same size) if reliability was the goal? I don't have a problem with backup requiring more space, but if losing those file makes failover more difficult, those shouldn't be [only] on the local host, but elsewhere, otherwise the most common need for a failover (host failure, maybe the second most common, after service maintenance) will not work.

Maybe those files should be sent elsewhere AND backed up? Again, I don't know anything about the internals, I am just making an assumption based on what I see on backup size.

ABran-WMF triaged this task as Medium priority.Dec 8 2025, 4:43 PM
ABran-WMF moved this task from Incoming to Work in Progress on the collaboration-services board.
LSobanski assigned this task to ABran-WMF.Dec 8 2025, 4:45 PM
ABran-WMF closed this task as Resolved.Dec 11 2025, 9:05 AM

I think my initial assumption was misguided, I was mistaken on the fileset Bacula targets.

Given:

In T406762#11423824, @Jelto wrote:

On the ReposEqiad bacula backend there are mostly ~10GB backups with a weekly bigger backup: https://grafana.wikimedia.org/d/413r2vbWk/bacula?orgId=1&from=2025-10-09T20:46:12.816Z&to=2025-12-02T13:50:27.684Z&timezone=utc&var-site=eqiad&var-job=gerrit1003.wikimedia.org-Hourly-Tue-ReposEqiad-gerrit-repo-data&viewPanel=panel-10

On the previous productionEqiad bacula backend the size was similar: https://grafana.wikimedia.org/d/413r2vbWk/bacula?orgId=1&from=2025-06-05T14:07:21.628Z&to=2025-10-09T12:41:20.566Z&timezone=utc&var-site=eqiad&var-job=gerrit1003.wikimedia.org-Hourly-Fri-productionEqiad-gerrit-repo-data&viewPanel=panel-10

I think we can assume that this is not an unusual pattern. And given:

In T406762#11410132, @jcrespo wrote:

These are the top files by size:

cumin2024@db1213.eqiad.wmnet[bacula9]> select Name, lstat_size(LStat) FROM File JOIN Filename USING(FilenameId) where JobId=666670 ORDER BY lstat_size(LStat) DESC LIMIT 15;
+-----------------------------+-------------------+
| Name                        | lstat_size(LStat) |
+-----------------------------+-------------------+
| git_file_diff.h2.db         |        2925682688 |
| comment_context.h2.db       |        1925754880 |
| account_patch_reviews.h2.db |        1427044352 |
| gerrit_file_diff.h2.db      |        1319897088 |
| mergeability.h2.db          |        1156052992 |
| diff_summary.h2.db          |        1086640128 |
| diff_intraline.h2.db        |         797939712 |
| conflicts.h2.db             |         787589120 |
| web_sessions.h2.db          |         705048576 |
| change_kind.h2.db           |         637464576 |
| git_modified_files.h2.db    |         349911040 |
| modified_files.h2.db        |         238133248 |
| master                      |          69714340 |
| accounts.h2.db              |          62945280 |
| persisted_projects.h2.db    |          30328832 |
+-----------------------------+-------------------+
15 rows in set (0.071 sec)

We can see that these files are cache dbs. According to Gerrit's documentation on caches, some are expensive to compute and their computation is triggered on demand by the UI. We're running replicas with no UI enabled, which prevents the computation of these caches. @Dzahn, @hashar, are you OK with my interpretation?

In T411583#11427381, @jcrespo wrote:

Question: I don't know the details of how are things are organized for resilience, but wouldn't the other host should have those files (and thus, backup would frequently have the same size) if reliability was the goal? I don't have a problem with backup requiring more space, but if losing those file makes failover more difficult, those shouldn't be [only] on the local host, but elsewhere, otherwise the most common need for a failover (host failure, maybe the second most common, after service maintenance) will not work.

Maybe those files should be sent elsewhere AND backed up? Again, I don't know anything about the internals, I am just making an assumption based on what I see on backup size.

Those files are not critical for a failover. With the current failover process, they are transferred to the new primary host. Given that Gerrit's compaction is not idempotent, I'm not sure we'd be able to use them if we just copied them without the rest of the data directory structure.

The next iteration of the process is aiming to avoid using rsync altogether, given those files sizes, we would probably benefit from triggering a cache creation on the promoted instance before setting it to read-write mode.

Since the backup size is not dramatically increasing over time, I'm marking this task as resolved. Feel free to reopen if needed!

jcrespo added a comment.Dec 11 2025, 9:12 AM

Those files are not critical for a failover. With the current failover process, they are transferred to the new primary host. Given that Gerrit's compaction is not idempotent, I'm not sure we'd be able to use them if we just copied them without the rest of the data directory structure.

Could you tell me how you do a failover if the primary host is down and unavailable? That is my main concern here. If they have to be transferred to work, but are not backed up (do not exist?) on the secondary host, that is a flaw on the backups. If they are not important for the secondary host to exist and work, then they should not be backed up (because they are not important). Either one or the other, or I am not understanding your logic.

jcrespo added a comment.Dec 11 2025, 9:13 AM

With the current failover process

That's a switchover process, not a failover process, hence my point/question.

jcrespo added a comment.Dec 11 2025, 9:15 AM

Alternatively, if they are "good to backup just in case", but not "critical", they should be backed up with the normal schedule (daily instead of hourly).

jcrespo reopened this task as Open.Dec 11 2025, 9:18 AM
ABran-WMF added a comment.Dec 11 2025, 9:25 AM
In T411583#11450494, @jcrespo wrote:

With the current failover process

That's a switchover process, not a failover process, hence my point/question.

I see where I was unclear, sorry about this!

The failover process is derivative from the one I've linked, but is not really documented yet.

In T411583#11450493, @jcrespo wrote:

Those files are not critical for a failover. With the current failover process, they are transferred to the new primary host. Given that Gerrit's compaction is not idempotent, I'm not sure we'd be able to use them if we just copied them without the rest of the data directory structure.

Could you tell me how you do a failover if the primary host is down and unavailable? That is my main concern here. If they have to be transferred to work, but are not backed up (do not exist?) on the secondary host, that is a flaw on the backups. If they are not important for the secondary host to exist and work, then they should not be backed up (because they are not important). Either one or the other, or I am not understanding your logic.

Good question! In the context of a failover, where the primary instance is unavailable for some reason, we would not have to transfer or restore anything as long as one of the replicas is still available.
The main focus here would be to mend Puppet and DNS so they point to the failover primary instance.

If we needed to restore from a backup, that would mean either we have no more Gerrit instance, or the data has somehow been corrupted.
In both context, I think it's best to have the cache DBs on par with the dataset that they are caching.

In T411583#11450496, @jcrespo wrote:

Alternatively, if they are "good to backup just in case", but not "critical", they should be backed up with the normal schedule (daily instead of hourly).

Given what I mention above, I'd be curious to know what @Dzahn @hashar think

@jcrespo please let me know if there is anything unclear!

ABran-WMF moved this task from Work in Progress to Awaiting Input on the collaboration-services board.Dec 11 2025, 9:26 AM
jcrespo added a comment.Dec 11 2025, 9:31 AM

So my takeaway is (simplifying):

  1. the data is critical and backups will be used for recovery in case of a failover, so it needs a full backup. Backup recovery cannot be ensured to be recovered properly.
  2. complete backups cannot be taken from replicas because they are not "complete backups"
  3. in the future, to relay less on backups, those files will be synced to the replicas

Ok- I don't think this is an ideal situation -it seems if primary gets corrupted of bad we will lose a lot of data-, but this ticket can resolved now.

ABran-WMF mentioned this in T412779: Gerrit failover process.Dec 16 2025, 6:21 AM
ABran-WMF closed this task as Resolved.Dec 16 2025, 7:20 AM

marking this as resolved, feel free to reopen if needed

Dzahn added a comment.Dec 16 2025, 3:05 PM

My thought here is just that the O'Reilly SRE book reminded me of "data recovery plans should not rely on replication".

ABran-WMF awarded a token.Dec 17 2025, 12:57 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits