Change 911363 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] gerrit: make the lfs data path configurable

https://gerrit.wikimedia.org/r/911363

Change 911363 merged by Dzahn:

[operations/puppet@production] gerrit: make the lfs data path configurable

https://gerrit.wikimedia.org/r/911363

I updated the code so that we have a new path for lfs data under /srv/gerrit/data, as suggested by hashar. It exists (as an empty dir) on all hosts now and on the new machine, gerrit1003, it will already use the new path. Also the path has been added to Bacula backups.

Now the lfs_path is a class parameter and can be set / overridden in Hiera.

So once we have switched to that we just need to move data on gerrit2002 (gerrit-replica) and decom gerrit1001 and this will be done.

Mentioned in SAL (#wikimedia-operations) [2023-04-25T21:17:02Z] <mutante> gerrit1003 - mv /srv/gerrit/plugins/lfs /srv/gerrit/data/ T333143 T326368

Mentioned in SAL (#wikimedia-operations) [2023-04-25T21:19:20Z] <mutante> gerrit1003 - chown -R gerrit2:gerrit2 /srv/gerrit T333143 T326368

Change 920765 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] gerrit: make new lfs path the default and clean up

https://gerrit.wikimedia.org/r/920765

I think after the patch above is merged we might be able to close this.

Change 908617 abandoned by Hashar:

[operations/puppet@production] gerrit: stop managing /srv/gerrit/plugins/lfs

Reason:

https://gerrit.wikimedia.org/r/908617

Change 920765 merged by Dzahn:

[operations/puppet@production] gerrit: remove lfs_dir parameter, use hardcoded new default

https://gerrit.wikimedia.org/r/920765

Ok, definitely doesn't seem "High" to me though since there is lots of space, no more lfs data in / and no more separate partition for /var/lib/gerrit2.

If we wanted to move even more to /srv/ than lfs I wish we had done that on the new server just recently.

Change 908604 merged by Jbond:

[operations/puppet@production] gerrit: remove duplicate $gerrit_site definition

https://gerrit.wikimedia.org/r/908604

The / partition was filing again due to firewall logs (T371951) and I thought we had Gerrit moved out of the / partition.

The caches and search indices have doubled since last year ( T333143#8935649 above):

$ du -hs /var/lib/gerrit2/review_site/{cache,index}
14G	/var/lib/gerrit2/review_site/cache
16G	/var/lib/gerrit2/review_site/index

Mentioned in SAL (#wikimedia-releng) [2025-03-10T13:47:44Z] <hashar> gerrit: removed leftover empty directory /srv/gerrit/plugins/lfs. Data have been migrated to /srv/gerrit/plugins/lfs as part of moving Gerrit data out of /. See T333143

In T333143#10618795, @Stashbot wrote:

gerrit: removed leftover empty directory /srv/gerrit/plugins/lfs. Data have been migrated to /srv/gerrit/plugins/lfs

The new path is /srv/gerrit/data/lfs.

modules/gerrit/templates/lfs.config.erb:    directory = /srv/gerrit/data/lfs

This has caused disk space issues again in T423027. This task is already tagged with Sustainability (Incident Followup) .

update April 2026

summary:

A large chunk of data has already been moved into /srv/ in the past; just not the entire /var/lib/gerrit.

Significant growth in usage since this ticket was created. (There might be periodical cleanup though.)

Still enough on /srv to comfortably move /var/lib/gerrit.

But this can't simply be moved to /srv/gerrit because that already exists as well.

Suggested run book to move remaining data:

prep:

• mkdir /srv/gerrit/site_path (we need/should use some new directory and this is what that is, /var/lib/gerrit (or previously /var/lib/gerrit2 is the default Gerrit Site Path.

• pre-rsync data from /var/lib/gerrit/ to /srv/gerrit/site_path/

migrate:

• stop gerrit

• rsync data from /var/lib/gerrit/ to /srv/gerrit/site_path/ one more time

• mv /var/lib/gerrit /srv/gerrit/var-lib-gerrit-backup (just in case, but don't forget it forever, /var/lib/gerrit needs to move out of the way though, or we simply "mv /var/lib/gerrit /srv/gerrit/site_path" and forget rsync!?)

• mount --bind /srv/gerrit/site_path /var/lib/gerrit (alternative: ln -s /srv/gerrit/site_path /var/lib/gerrit)

• start gerrit

?

We might want to stop puppet first and have a merged change that will update httpd's config

In T333143#11817082, @Dzahn wrote:

Suggested run book to move remaining data:

prep:

• mkdir /srv/gerrit/site_path (we need/should use some new directory and this is what that is, /var/lib/gerrit (or previously /var/lib/gerrit2 is the default Gerrit Site Path.

• pre-rsync data from /var/lib/gerrit/ to /srv/gerrit/site_path/

migrate:

• stop gerrit

• rsync data from /var/lib/gerrit/ to /srv/gerrit/site_path/ one more time

• mv /var/lib/gerrit /srv/gerrit/var-lib-gerrit-backup (just in case, but don't forget it forever, /var/lib/gerrit needs to move out of the way though, or we simply "mv /var/lib/gerrit /srv/gerrit/site_path" and forget rsync!?)
• mount --bind /srv/gerrit/site_path /var/lib/gerrit (alternative: ln -s /srv/gerrit/site_path /var/lib/gerrit)

• start gerrit

?

I think the symlink will be good enough for that, until we apply the puppet patch to reflect the migration, and then we can discard it. Otherwise the approach looks good to me, thanks for writing it down!

I think we'll also need to update the /etc/gerrit link:

arnaudb@gerrit2003:~ $ ls -l /etc/gerrit
lrwxrwxrwx 1 root root 31 Nov  8  2024 /etc/gerrit -> /var/lib/gerrit/review_site/etc

to make Gerrit start properly after migrating

Change #1270774 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/puppet@production] gerrit: migrate gerrit_site away from root partition

https://gerrit.wikimedia.org/r/1270774

In T333143#11817082, @Dzahn wrote:

Suggested run book to move remaining data:

prep:

• mkdir /srv/gerrit/site_path (we need/should use some new directory and this is what that is, /var/lib/gerrit (or previously /var/lib/gerrit2 is the default Gerrit Site Path.

• pre-rsync data from /var/lib/gerrit/ to /srv/gerrit/site_path/

migrate:

• stop gerrit

• rsync data from /var/lib/gerrit/ to /srv/gerrit/site_path/ one more time

• mv /var/lib/gerrit /srv/gerrit/var-lib-gerrit-backup (just in case, but don't forget it forever, /var/lib/gerrit needs to move out of the way though, or we simply "mv /var/lib/gerrit /srv/gerrit/site_path" and forget rsync!?)

• mount --bind /srv/gerrit/site_path /var/lib/gerrit (alternative: ln -s /srv/gerrit/site_path /var/lib/gerrit)

• start gerrit

?

Thank you @Dzahn for the detailed runbook. I made the commands a bit more specific (especially to preserve file permissions with rsync). So the actual commands would be:

Prep:

• sudo mkdir /srv/gerrit/site_path && sudo chown gerrit:gerrit /srv/gerrit/site_path
• sudo rsync -a /var/lib/gerrit/ /srv/gerrit/site_path/

Migrate:

• sudo disable-puppet "migrate gerrit data out of root partition - T333143 "
• sudo systemctl stop gerrit.service
• sudo rsync -a /var/lib/gerrit/ /srv/gerrit/site_path/
• sudo mv /var/lib/gerrit /srv/gerrit/var-lib-gerrit-backup
• sudo ln -s /srv/gerrit/site_path /var/lib/gerrit
• merge puppet patch ?
• sudo enable-puppet "migrate gerrit data out of root partition - T333143 "
• sudo run-puppet-agent
• sudo systemctl restart gerrit.service (probably done by puppet already)

cleanup:

• sudo unlink /var/lib/gerrit
• sudo rm -rf /srv/gerrit/var-lib-gerrit-backup

feel free to change/amend any command in case I forgot something. I'm not fully sure when the puppet patch above should be merged.

Change #1270863 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/cookbooks@master] gerrit: update sync-instances cookbook

https://gerrit.wikimedia.org/r/1270863

I've tested the migration snippet on the spare instance gerrit2002. Beside fixing a typo there is a problem with puppet re-creating the /var/lib/gerrit folder:

gerrit2002 puppet-agent[2107008]: (/Stage[main]/Gerrit/File[/var/lib/gerrit]/ensure) ensure changed 'link' to 'directory' (corrective)

So puppet is removing the symlink and creates a new empty folder with some puppet-managed files. I'll try setting profile::gerrit::gerrit_site: /srv/gerrit/site_path for gerrit2002 and see if that fixes the issue. Otherwise we might need to use the bind mount approach or write some more puppet code.

Change #1272612 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/puppet@production] gerrit: migrate data ways from /var/lib/gerrit on gerrit2002

https://gerrit.wikimedia.org/r/1272612

Change #1272612 merged by Jelto:

[operations/puppet@production] gerrit: migrate data ways from /var/lib/gerrit on gerrit2002

https://gerrit.wikimedia.org/r/1272612

Change #1272718 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/puppet@production] gerrit: make daemon_user_dir configurable and set it to /srv for gerrit2002

https://gerrit.wikimedia.org/r/1272718

I did another migration attempt and puppet keeps re-creating the /var/lib/gerrit even when profile::gerrit::gerrit_site: /srv/gerrit/site_path is set. This happenes because it's hardcoded in init.pp:

$daemon_user_dir = "/var/lib/${daemon_user}"

I opened https://gerrit.wikimedia.org/r/c/operations/puppet/+/1272718 to make the daemon_user_dir configurable so we can set it on a per-host basis.

We can cleanup some of that code after the migration.

Change #1272718 merged by Jelto:

[operations/puppet@production] gerrit: make daemon_user_dir configurable and set it to /srv for gerrit2002

https://gerrit.wikimedia.org/r/1272718

Host gerrit2002.wikimedia.org rebooted by jelto@cumin1003 with reason: maintenance reboot

Change #1273449 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/puppet@production] gerrit: migrate data gerrit1003 to /srv/gerrit

https://gerrit.wikimedia.org/r/1273449

Change #1273683 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/puppet@production] gerrit: migrate gerrit2003 data to /srv/gerrit

https://gerrit.wikimedia.org/r/1273683

There is an active incinga alert for the migrated spare host:

PROCS CRITICAL: 0 processes with regex args '^/usr/lib/jvm/java-17-openjdk-amd64/bin/java .*-jar /srv/gerrit/site_path/review_site/bin/gerrit.war daemon -d /srv/gerrit/site_path/review_site'

Indeed the java process is still pointing to the old location:

/usr/lib/jvm/java-17-openjdk-amd64/bin/java -Xmx64g -Xms64g ... -XX:+ExitOnOutOfMemoryError -XX:HeapDumpPath=/srv/gerrit -jar /var/lib/gerrit/review_site/bin/gerrit.war daemon -d /var/lib/gerrit/review_site --enable-httpd --replica

I'll try to fix this before migrating the replica and the production instance.

I think the systemd service is configured in gerrit.systemd.erb.
However on the gerrit spare host I see two systemd unit files:

• /etc/systemd/system/gerrit.service with the old path
• /lib/systemd/system/gerrit.service with the new path

The alert mentioned above is resolved after removing the manually created systemd unit file in /etc/systemd/system/gerrit.service on gerrit2002. All gerrit hosts use the puppet-managed unit file again.

Change #1273449 merged by Jelto:

[operations/puppet@production] gerrit: migrate data gerrit1003 to /srv/gerrit

https://gerrit.wikimedia.org/r/1273449

The replica host gerrit1003 has been migrated to /srv/gerrit.

Full runbook of commands:

Migrate:

• sudo disable-puppet "migrate gerrit data out of root partition - T333143 "
• sudo systemctl stop gerrit.service
• sudo rsync -a /var/lib/gerrit/ /srv/gerrit/site_path/
• sudo mv /var/lib/gerrit /srv/gerrit/var-lib-gerrit-backup
• sudo ln -s /srv/gerrit/site_path /var/lib/gerrit
• sudo killall -u gerrit && sudo /usr/sbin/usermod -d /srv/gerrit gerrit (puppet can not update the folder for an existing user properly, because it starts other gerrit services again during the puppet run).
• merge puppet patch to set profile::gerrit::gerrit_site: /srv/gerrit/site_path and daemon_user_dir: /srv/gerrit for the specific host
• sudo enable-puppet "migrate gerrit data out of root partition - T333143 "
• sudo run-puppet-agent

A reboot does not seem to be necessary. This reduced file system usage for the root partition from over 60% to under 30%.

Runbook for the production host:

Migrate:

• sudo cookbook sre.hosts.downtime --minutes 30 -r "Gerrit maintenance" -t T333143 gerrit2003.wikimedia.org
• sudo cookbook sre.hosts.downtime --minutes 30 -r "Gerrit maintenance" -t T333143 --force gerrit.discovery.wmnet
• sudo disable-puppet "migrate gerrit data out of root partition - T333143 "
• sudo systemctl stop gerrit.service
• sudo rsync -a /var/lib/gerrit/ /srv/gerrit/site_path/
• sudo mv /var/lib/gerrit /srv/gerrit/var-lib-gerrit-backup
• sudo ln -s /srv/gerrit/site_path /var/lib/gerrit
• sudo killall -u gerrit && sudo /usr/sbin/usermod -d /srv/gerrit gerrit (puppet can not update the folder for an existing user properly, because it starts other gerrit services again during the puppet run).
• merge puppet patch to set profile::gerrit::gerrit_site: /srv/gerrit/site_path and daemon_user_dir: /srv/gerrit for the specific host
• sudo enable-puppet "migrate gerrit data out of root partition - T333143 "
• sudo run-puppet-agent

Icinga downtime and Alertmanager silence (ID=a9ab2e33-702e-4e70-a1fb-3a2b8f7b8c2b) set by jelto@cumin1003 for 0:35:00 on 1 host(s) and their services with reason: Gerrit maintenance

gerrit2003.wikimedia.org

Change #1273683 merged by Jelto:

[operations/puppet@production] gerrit: migrate gerrit2003 data to /srv/gerrit

https://gerrit.wikimedia.org/r/1273683

Mentioned in SAL (#wikimedia-operations) [2026-04-23T06:04:59Z] <jelto> start gerrit2003 maintenance - T333143

Mentioned in SAL (#wikimedia-operations) [2026-04-23T06:28:05Z] <jelto> gerrit2003 maintenance finished - T333143

Data on gerrit2003 production gerrit has also been moved to /srv/gerrit.

Disk usage on / is down from 60% to 30%.

Change #1270774 merged by Jelto:

[operations/puppet@production] gerrit: migrate gerrit_site away from root partition

https://gerrit.wikimedia.org/r/1270774

All hosts are migrated and cleanup has happened on gerrit-replica and gerrit-spare. I'll wait until next week for the gerrit production host cleanup (removing the symlink and the temporary backup folder /srv/gerrit/var-lib-gerrit-backup).

Change #1270863 merged by jenkins-bot:

[operations/cookbooks@master] gerrit: update sync-instances cookbook

https://gerrit.wikimedia.org/r/1270863

Change #1284634 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/deployment-charts@master] miscweb: use url-downloader proxy for wmf-navigator

https://gerrit.wikimedia.org/r/1284634

Change #1284634 merged by jenkins-bot:

[operations/deployment-charts@master] miscweb: use url-downloader proxy for wmf-navigator, add service listener

https://gerrit.wikimedia.org/r/1284634