From

It looks like memory usage is slowing growing since mid august

That might be related to the deployment of git-protocol-v2 on clients. For CI that was T256844 done on August 18th and 19th:

August 18th, most jobs, 7d25340b0024631c194a06afa6d89f89a209ec99
August 19th, Quibble jobs, a566f054effdd9f46b5b9a3fa4031844fe0efd46

Which might be triggering an issue of some sort in jgit and our Gerrit deployment comes with jgit v5.8.1.202007141445-r.

I guess we can check the memory usage again in a week / two weeks at https://gerrit.wikimedia.org/r/monitoring?part=graph&graph=usedMemory&period=month .

I went to take a heap dump of current memory usage without objects pending finalization/garbage collection (dump:live):

sudo -u gerrit2 jmap -dump:live,format=b,file=/var/lib/gerrit-202009170755.hprof <pid of java here>

Which I have moved in my home dir. Ran jhat -stack false -refs false gerrit-202009170755.hprof which takes a while and boot a web server on port 7000.

On my local machine: ssh -C -L 8080:ip6-localhost:7000 gerrit1001.wikimedia.org and then http://ip6-localhost:8080/

Some metrics:

Instance Counts for All Classes (excluding platform)

2237744 instances of class org.eclipse.jgit.lib.ObjectId
2128766 instances of class org.eclipse.jgit.lib.ObjectIdRef$PeeledNonTag
735294 instances of class org.eclipse.jetty.util.thread.Locker
735294 instances of class org.eclipse.jetty.util.thread.Locker$Lock
735283 instances of class org.eclipse.jetty.server.session.Session
735283 instances of class org.eclipse.jetty.server.session.Session$SessionInactivityTimer
735283 instances of class org.eclipse.jetty.server.session.Session$SessionInactivityTimer$1
735283 instances of class org.eclipse.jetty.server.session.SessionData
198756 instances of class org.eclipse.jgit.internal.storage.file.ByteArrayWindow
198756 instances of class org.eclipse.jgit.internal.storage.file.WindowCache$Entry
198756 instances of class org.eclipse.jgit.internal.storage.file.WindowCache$SoftRef
172260 instances of class org.h2.value.ValueJavaObject
163182 instances of class com.codahale.metrics.WeightedSnapshot$WeightedSample
146256 instances of class org.h2.value.ValueTimestamp
136314 instances of class [Lorg.h2.value.Value;
134132 instances of class com.googlecode.javaewah.EWAHCompressedBitmap
134132 instances of class com.googlecode.javaewah.LongArray
134132 instances of class com.googlecode.javaewah.RunningLengthWord
130280 instances of class org.eclipse.jgit.internal.storage.file.BasePackBitmapIndex$StoredBitmap
113191 instances of class org.h2.result.Row
82653 instances of class org.eclipse.jgit.diff.Edit
70534 instances of class org.eclipse.jgit.revwalk.RevCommit
66529 instances of class [Lorg.eclipse.jgit.revwalk.RevCommit;
64096 instances of class org.eclipse.jgit.revwalk.RevTree
57190 instances of class org.eclipse.jgit.internal.storage.file.BasePackBitmapIndex$XorCompressedBitmap
...

Instance Counts for All Classes (Including platform)

4859984 instances of class java.lang.String
3839858 instances of class [C
3830534 instances of class java.util.concurrent.ConcurrentHashMap$Node
2237744 instances of class org.eclipse.jgit.lib.ObjectId
2128766 instances of class org.eclipse.jgit.lib.ObjectIdRef$PeeledNonTag
1830871 instances of class [B
750976 instances of class java.util.concurrent.ConcurrentHashMap
750486 instances of class java.util.concurrent.atomic.AtomicReference
745988 instances of class [I
743997 instances of class java.util.concurrent.locks.ReentrantLock$NonfairSync
742434 instances of class [Ljava.util.concurrent.ConcurrentHashMap$Node;
742425 instances of class java.util.concurrent.locks.ReentrantLock
735428 instances of class java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject
735294 instances of class org.eclipse.jetty.util.thread.Locker
735294 instances of class org.eclipse.jetty.util.thread.Locker$Lock
735283 instances of class org.eclipse.jetty.server.session.Session
735283 instances of class org.eclipse.jetty.server.session.Session$SessionInactivityTimer
735283 instances of class org.eclipse.jetty.server.session.Session$SessionInactivityTimer$1
735283 instances of class org.eclipse.jetty.server.session.SessionData
229023 instances of class java.util.LinkedHashMap$Entry
209390 instances of class java.util.HashMap$Node
198756 instances of class org.eclipse.jgit.internal.storage.file.ByteArrayWindow
198756 instances of class org.eclipse.jgit.internal.storage.file.WindowCache$Entry
198756 instances of class org.eclipse.jgit.internal.storage.file.WindowCache$SoftRef
172260 instances of class org.h2.value.ValueJavaObject
166482 instances of class [J
163533 instances of class java.util.concurrent.ConcurrentSkipListMap$Node
163230 instances of class java.lang.Double
163182 instances of class com.codahale.metrics.WeightedSnapshot$WeightedSample
146256 instances of class org.h2.value.ValueTimestamp
136314 instances of class [Lorg.h2.value.Value;
134132 instances of class com.googlecode.javaewah.EWAHCompressedBitmap
134132 instances of class com.googlecode.javaewah.LongArray
134132 instances of class com.googlecode.javaewah.RunningLengthWord
130280 instances of class org.eclipse.jgit.internal.storage.file.BasePackBitmapIndex$StoredBitmap
113191 instances of class org.h2.result.Row
99759 instances of class [Ljava.lang.Object;
82653 instances of class org.eclipse.jgit.diff.Edit
...

Heap Histogram All Classes (excluding platform)

you might also try something like https://heaphero.io/ to see if there are any additional insights it can give you. I think I'm still paying for fastthread.io (which is the same company -- may be useful).

Change 628338 had a related patch set uploaded (by Hashar; owner: Hashar):
[operations/puppet@production] gerrit: dump heap on out of memory error

https://gerrit.wikimedia.org/r/628338

In T263008#6471859, @thcipriani wrote:

you might also try something like https://heaphero.io/ to see if there are any additional insights it can give you. I think I'm still paying for fastthread.io (which is the same company -- may be useful).

Nice they have bunch of nice utilities. There are a couple issues though: the heap dump might well contain a bunch of credentials and personal information AND it was 5GBytes when I took one without the objects that were ready for garbage collection :]

Change 628338 merged by Dzahn:
[operations/puppet@production] gerrit: dump heap on out of memory error

https://gerrit.wikimedia.org/r/628338

Mentioned in SAL (#wikimedia-operations) [2020-09-18T15:09:07Z] <mutante> restarting gerrit service to apply gerrit::628338 to make it dump heap if out of memory (T263008)

Here is the memory report for the last 15 days:

We had roughly 4GBytes heap mean usage increase over the last few days. Does not seem too concerning so far.

Based on https://gerrit.wikimedia.org/r/monitoring?part=heaphisto out of 20GBytes used:

Heap Classes: 10,523, Instances: 225,343,723, Kilo-Bytes: 20,459,013

And after executing the garbage collector:

Garbage collector executed (5,703,919 Kb freed, 16,478,791 Kb still used)

Heap Classes: 8,819, Instances: 207,771,485, Kilo-Bytes: 16,613,165

So it reclaimed 2GBytes of byte[] and 3.7G of other things.

And eventually 15 days after I triggered a full gc, we had very hight memory usage and @thcipriani had Gerrit restarted due to gc trashing:

17:43 	<thcipriani> 	restarting gerrit due to gc thrashing

Could it be that we had full gc disabled somehow?

Gerrit tuning infos: https://ctf-dev-environment-vagrant.s3.amazonaws.com/Gerrit-Performance-Tuning-Cheat-Sheet.pdf

We probably want to look at jgit config:

[core]
    packedGitOpenFiles = <%= @git_open_files %>  # 20000
    packedGitLimit = 4g
    packedGitWindowSize = 16k
    streamFileThreshold = 100m
    useRecursiveMerge = true

The packedGitOpenFiles got raised from 6k to 20k three years ago due to T168360: Gerrit constantly throws HTTP 500 error when reviewing patches (due to "Too many open files").

In T263008#6565092, @hashar wrote:

Gerrit tuning infos: https://ctf-dev-environment-vagrant.s3.amazonaws.com/Gerrit-Performance-Tuning-Cheat-Sheet.pdf

We probably want to look at jgit config:

modules/gerrit/templates/gerrit.config.erb

[core]
    packedGitOpenFiles = <%= @git_open_files %>  # 20000
    packedGitLimit = 4g

^ packedGitLimit is, I would guess, where we're running into issues. I last looked at this about a year ago in T221026#5113741

I have seen recommendations from jGit developers running git to have this be half the size of the heap. My hunch is that since we started using protocol v2 we're seeing more pack files than we had previously. We've also added 555 projects between April 2019 (when I last touched this) and now. Doubling this seems like a good starting point. Possibly also increasing overall heap size to accommodate the extra space without sacrificing anything else using heap.

Mentioned in SAL (#wikimedia-operations) [2020-11-03T21:16:43Z] <hashar> Gerrit: triggering java garbage collection # T263008

Found some more settings mentioned at http://gitblit.com/setup_scaling.html , they come from a presentation by Shawn Pearce ( page 18 of https://www.eclipsecon.org/2013/sites/eclipsecon.org.2013/files/Scaling%20Up%20JGit%20-%20EclipseCon%202013.pdf ).

# -Xmx8g # gobs of JVM heap
# for `jgit daemon` or gerrit.config
[core]
    packedGitLimit = 4g # around 50% of JVM heap
    packedGitOpenFiles = 8192 # don't forget to configure ulimit
    streamFileThreshold = 2047m # avoids pathological inflate
    deltaBaseCacheLimit = 50m

# applies to `jgit gc`
[pack]
    bigFileThreshold = 20m # don't delta compress big binaries
    indexVersion = 2 # use index v2

So essentially, yes we will have to finely tweak jgit via settings at https://gerrit-review.googlesource.com/Documentation/config-gerrit.html#core Some hints:

core.streamFileThreshold

Largest object size, in bytes, that JGit will allocate as a contiguous byte array. Any file revision larger than this threshold will have to be streamed, typically requiring the use of temporary files under '$GIT_DIR/objects' to implement pseudo-random access during delta decompression.

Servers with very high traffic should set this to be larger than the size of their common big files. For example a server managing the Android platform typically has to deal with ~10-12 MiB XML files, so 15 m would be a reasonable setting in that environment. Setting this too high may cause the JVM to run out of heap space when handling very big binary files, such as device firmware or CD-ROM ISO images.

Defaults to 25% of the available JVM heap, limited to 2g.

Common unit suffixes of 'k', 'm', or 'g' are supported.

So if I read it properly, we could have pack files up to 2G. Looking at lsof there are a few large ones:

But I guess we want to support mediawiki/core @ 624MBytes

core.repositoryCacheExpireAfter

Time an unused repository should expire and be evicted from the repository cache.

Values can be specified using standard time unit abbreviations (ms, sec, min, etc.).
Default is 1 hour.

Maybe we can lower that one to force eviction. But what would be really nice, is to have some metrics about jgit daemon :/

That happened on the Gerrit replica gerrit2001:

Nov 07 13:43:35 gerrit2001 java[30197]: java.lang.OutOfMemoryError: Java heap space
Nov 07 13:43:35 gerrit2001 java[30197]: Dumping heap to /srv/gerrit/java_pid30197.hprof ...
Nov 07 13:47:02 gerrit2001 java[30197]: Heap dump file created [35616147146 bytes in 206.962 secs]

Since we now run Gerrit with:

We have a nice 34GB heap dump to inspect on gerrit2001 at /srv/gerrit/java_pid30197.hprof.

Looks like I forgot to flag monitoring as a requirement to solve this one: T184086 . Without metrics, we dont quite know what is going on in Gerrit internals.

Happened again 2020-12-18:

The exact same slow pill up of heap usage :-\

One of the things I anticipated was that the jgit block cache would fill up, but it seems like that's not the case. That is, it seems like the closer we get to running out of memory the more the jvm ditches jgit block cache space. This likely means slower operations, a bigger stack up of threads, and more memory flailing.

Mentioned in SAL (#wikimedia-operations) [2020-12-22T08:52:02Z] <hashar> gerrit: running jhat heap analyzer on gerrit2001 # T263008

There seems to be a memory leak with jetty sessions:

There is really no reason for having a million of jetty session being held.

There are some informations at https://github.com/dropwizard/dropwizard/issues/2455

And 19 hours after P13625:

hashar@gerrit2001:~$ sudo -u gerrit2 jcmd 2347 GC.class_histogram|grep -i jetty
   4:      10512257     1261470840  org.eclipse.jetty.server.session.SessionData
   7:      10512257      925078616  org.eclipse.jetty.server.session.Session
  10:      10512257      504588336  org.eclipse.jetty.server.session.Session$SessionInactivityTimer$1
  13:      10512267      336392544  org.eclipse.jetty.util.thread.Locker
  14:      10512257      336392224  org.eclipse.jetty.server.session.Session$SessionInactivityTimer
  17:      10512267      252294408  org.eclipse.jetty.util.thread.Locker$Lock

500k more objects / 56 MB more just for org.eclipse.jetty.server.session.SessionData. So it is slowly leaking memory.

Filed upstream https://bugs.chromium.org/p/gerrit/issues/detail?id=13858

It is neither Java 8 or Gerrit 3.2 related as far as I can tell, will drop the sub tasks.

Reported to Upstream and we have found the root cause. It is being worked on :]

For Security-Team that is a potential denial of service in Gerrit. After a few millions of git operations over https, the Java VM runs out of HEAP memory. Though surely millions of requests in a short time frame might be noticed.

Workaround: restart Gerrit instance to clear out the accumulated zombie objects.

I guess upstream will release a security release over the next few weeks.

I have restarted both of our Gerrit instances to clear out the memory leak.

I wrote a blog post summarizing this adventure: Blog Post: Tracking memory issue in a Java application