Page MenuHomePhabricator

Gerrit crashed due to out of Heap
Open, Needs TriagePublic

Description

Today Gerrit required a restart, it appears from the error log that it is due to heap exhaustion.

[2019-06-05 22:07:36,449] [Thread-24] ERROR com.google.gerrit.pgm.Daemon : Thread Thread-24 threw exception
java.lang.OutOfMemoryError: Java heap space
        at java.lang.Integer.valueOf(Integer.java:832)
        at sun.nio.ch.EPollPort$EventHandlerTask.poll(EPollPort.java:223)
        at sun.nio.ch.EPollPort$EventHandlerTask.run(EPollPort.java:268)
        at java.lang.Thread.run(Thread.java:748)
[2019-06-05 22:07:36,455] [HTTP-140393] WARN  org.eclipse.jetty.servlet.ServletHandler : Error for /r/mediawiki/extensions/LiquidThreads/info/refs
java.lang.OutOfMemoryError: Java heap space
[2019-06-05 22:07:36,455] [HTTP-140054] WARN  /r : Internal error during upload-pack from /srv/gerrit/git/operations/puppet.git
java.lang.OutOfMemoryError: Java heap space
[2019-06-05 22:07:36,455] [HTTP-139507] WARN  org.eclipse.jetty.servlet.ServletHandler : Error for /r/mediawiki/extensions/JsonConfig/info/refs
java.lang.OutOfMemoryError: Java heap space
[2019-06-05 22:07:36,454] [accounts NRT] ERROR com.google.gerrit.pgm.Daemon : Thread accounts NRT threw exception
java.lang.OutOfMemoryError: Java heap space
[2019-06-05 22:07:36,454] [HTTP-140373] WARN  org.eclipse.jetty.servlet.ServletHandler : Error for /r/mediawiki/extensions/Echo/info/refs
java.lang.OutOfMemoryError: Java heap space

Event Timeline

crusnov created this task.Jun 5 2019, 10:22 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 5 2019, 10:22 PM
CDanis added a subscriber: CDanis.Jun 5 2019, 10:32 PM

Some curious stuff in the monitoring data:

I have gc info from right when this happened: https://gceasy.io/my-gc-report.jsp?p=c2hhcmVkLzIwMTkvMDYvNS8tLWp2bV9nYy5nZXJyaXQubG9nLjcuY3VycmVudC0tMjMtMjgtNTE=&channel=WEB

Looks like there was a sudden spike in heap usage starting at 22:06:03, that triggered a full gc (causing a huge pause) at 22:06:53.

The first out of OutOfMemory error happened at 22:07:36:

thcipriani@cobalt:~$ grep -B1 -i 'heap' /var/log/gerrit/error_log | head -n2
[2019-06-05 22:07:36,449] [Thread-24] ERROR com.google.gerrit.pgm.Daemon : Thread Thread-24 threw exception
java.lang.OutOfMemoryError: Java heap space