Page MenuHomePhabricator

gerrit-replica is OOMing again, returning 502s on git clones
Closed, ResolvedPublic


Previously: T240763: gerrit-replica is returning 502 responses when trying to git clone, breaking libup

monthly memory usage:

gerrit-replica-memory-monthly.png (499×1 px, 22 KB)


gerrit-replica-memory.png (499×1 px, 21 KB)

there's definitely some kind of memory leak that needs to be addressed.

Event Timeline

Legoktm triaged this task as Unbreak Now! priority.Jan 25 2020, 4:37 AM
Legoktm created this task.
thcipriani claimed this task.

Here's a threaddump I managed to grab before restarting:

Looks like there was a blocked thread, but I'm unclear if that was due to memory issues that were already present.

Also there was a recent load/CPU spike -- this looks like Friday git gc activity

systemLoad.png (499×1 px, 47 KB)

Most likely this machine needs some tuning for its specific traffic patterns, nothing looks out of the ordinary, really, just a JVM that couldn't handle the traffic. Filed as: T243655: Tune gerrit-replica to be more resilient to the kind of traffic it gets