Move Gerrit data out of root partition
Open, MediumPublic
Actions

Assigned To

None

Authored By

	hashar
	Mar 27 2023, 12:02 PM

Description

During the Gerrit 3.5 upgrade, Gerrit caches stored under /var/lib/gerrit2 overflowed the root partition. As an immediate fix we have created a new dedicated partition and moved data there. The incident report https://wikitech.wikimedia.org/wiki/Incidents/2022-11-17_Gerrit_3.5_upgrade

From the incident analysis at T323262#8435688 we should relocate Gerrit installation from /var/lib/gerrit2/review_site to /srv/gerrit.

On gerrit1001 the transient /var/lib/gerrit2 partition should be removed.

Partition scheme for both hosts:

gerrit1001.wikimedia.org

Partition	Size	Used	Available
`/`	46G	14G	30G
`/srv`	314G	227G	71G
`/var/lib/gerrit2`	49G	17G	31G

gerrit2002.wikimedia.org

Partition	Size	Used	Available
`/`	73G	20G	50G
`/srv`	629G	71G	527G

Details

Subject	Repo	Branch	Lines +/-
gerrit: remove duplicate $gerrit_site definition	operations/puppet	production	+24 -11
gerrit: remove lfs_dir parameter, use hardcoded new default	operations/puppet	production	+3 -10
gerrit: stop managing /srv/gerrit/plugins/lfs	operations/puppet	production	+0 -1
gerrit: make the lfs data path configurable	operations/puppet	production	+7 -1
gerrit: add /srv/gerrit/data/lfs to dirs managed by puppet	operations/puppet	production	+2 -0
backup: add /srv/gerrit/data to fileset for gerrit repos	operations/puppet	production	+1 -1

Customize query in gerrit

Related Objects

Mentioned In: T326368: gerrit1003 service implementation task
T323262: gerrit1001 running out of space on /
Mentioned Here: T326368: gerrit1003 service implementation task
T323262: gerrit1001 running out of space on /

Event Timeline

hashar created this task.Mar 27 2023, 12:02 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 27 2023, 12:02 PM

hashar mentioned this in T323262: gerrit1001 running out of space on /.Mar 27 2023, 12:06 PM

jcrespo awarded a token.Mar 28 2023, 11:20 AM

LSobanski triaged this task as High priority.Apr 3 2023, 3:56 PM

LSobanski moved this task from Incoming to Backlog on the collaboration-services board.

Change 908604 had a related patch set uploaded (by Hashar; author: Hashar):

[operations/puppet@production] gerrit: remove duplicate $gerrit_site definition

https://gerrit.wikimedia.org/r/908604

gerritbot added a project: Patch-For-Review.Apr 13 2023, 5:46 PM

The LFS plugin stores the data under /srv/gerrit/plugins/lfs which will clash with the $GERRIT_SITE/plugins directory holding the jar/js plugins.

Change 908617 had a related patch set uploaded (by Hashar; author: Hashar):

[operations/puppet@production] gerrit: relocate LFS data

https://gerrit.wikimedia.org/r/908617

Change 911358 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] gerrit: add /srv/gerrit/data/lfs to dirs managed by puppet

https://gerrit.wikimedia.org/r/911358

Change 911358 merged by Dzahn:

[operations/puppet@production] gerrit: add /srv/gerrit/data/lfs to dirs managed by puppet

https://gerrit.wikimedia.org/r/911358

Change 911362 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] backup: add /srv/gerrit/data to fileset for gerrit repos

https://gerrit.wikimedia.org/r/911362

Change 911362 merged by Dzahn:

[operations/puppet@production] backup: add /srv/gerrit/data to fileset for gerrit repos

https://gerrit.wikimedia.org/r/911362

Change 911363 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] gerrit: make the lfs data path configurable

https://gerrit.wikimedia.org/r/911363

Change 911363 merged by Dzahn:

[operations/puppet@production] gerrit: make the lfs data path configurable

https://gerrit.wikimedia.org/r/911363

I updated the code so that we have a new path for lfs data under /srv/gerrit/data, as suggested by hashar. It exists (as an empty dir) on all hosts now and on the new machine, gerrit1003, it will already use the new path. Also the path has been added to Bacula backups.

Now the lfs_path is a class parameter and can be set / overridden in Hiera.

So once we have switched to that we just need to move data on gerrit2002 (gerrit-replica) and decom gerrit1001 and this will be done.

Mentioned in SAL (#wikimedia-operations) [2023-04-25T21:17:02Z] <mutante> gerrit1003 - mv /srv/gerrit/plugins/lfs /srv/gerrit/data/ T333143 T326368

Mentioned in SAL (#wikimedia-operations) [2023-04-25T21:19:20Z] <mutante> gerrit1003 - chown -R gerrit2:gerrit2 /srv/gerrit T333143 T326368

Change 920765 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] gerrit: make new lfs path the default and clean up

https://gerrit.wikimedia.org/r/920765

I think after the patch above is merged we might be able to close this.

Dzahn moved this task from Backlog to Work in Progress on the collaboration-services board.May 17 2023, 6:54 PM

Change 908617 abandoned by Hashar:

[operations/puppet@production] gerrit: stop managing /srv/gerrit/plugins/lfs

Reason:

https://gerrit.wikimedia.org/r/908617

Change 920765 merged by Dzahn:

[operations/puppet@production] gerrit: remove lfs_dir parameter, use hardcoded new default

https://gerrit.wikimedia.org/r/920765

LSobanski assigned this task to Dzahn.Jun 5 2023, 2:03 PM

As this ticket asks for " the transient /var/lib/gerrit2 partition should be removed" and what we have actually done is move the lfs data outside of it:

We can say that both current gerrit servers, gerrit1003 and gerrit2002, have no more separate /var/lib/gerrit2 partition.

/var/lib/gerrit2 is still 15G on gerrit1003 but it's simply part of the / partition which is 68% used with 23G available and it's not expected to grow much.

Meanwhile the lfs data and git repos are in /srv and it's 37% used with 379G available.

gerrit2002 has even much more space in both locations.

gerrit1001 will be shut down and is already not considered a gerrit server anymore.

So this should be resolved.

Reopening since this task is about relocating Gerrit from the root partition to /srv:

From the incident analysis at T323262#8435688 we should relocate Gerrit installation from /var/lib/gerrit2/review_site to /srv/gerrit.

The incident was the H2 database (stored in db directory) overflowing the root partition. There are other growing directories, notably the disk based caches in cache and Lucene search indices in index.

sudo du -m -d1 /var/lib/gerrit2/review_site
...
216	/var/lib/gerrit2/review_site/data
7358	/var/lib/gerrit2/review_site/cache
6973	/var/lib/gerrit2/review_site/index
859	/var/lib/gerrit2/review_site/db
15411	/var/lib/gerrit2/review_site

Ok, definitely doesn't seem "High" to me though since there is lots of space, no more lfs data in / and no more separate partition for /var/lib/gerrit2.

If we wanted to move even more to /srv/ than lfs I wish we had done that on the new server just recently.

The /var/lib/gerrit2/review_site/db directory is only 1 GB with all the reviews so far. There are 22GB free. That makes me think this will not be a real problem any time in the next decade or so. (?)

well, to be fair, the cache and index are a bit more:

root@gerrit1003:/var/lib/gerrit2/review_site# du -hs cache/
7.2G	cache/
root@gerrit1003:/var/lib/gerrit2/review_site# du -hs index/
6.9G	index/

Dzahn moved this task from Work in Progress to Backlog on the collaboration-services board.Jun 15 2023, 4:09 PM

Change 908604 merged by Jbond: