Page MenuHomePhabricator

gerrit-replica is OOMing again, returning 502s on git clones
Closed, ResolvedPublic

Description

Previously: T240763: gerrit-replica is returning 502 responses when trying to git clone, breaking libup

monthly memory usage:

yearly:

there's definitely some kind of memory leak that needs to be addressed.

Event Timeline

Legoktm triaged this task as Unbreak Now! priority.Jan 25 2020, 4:37 AM
Legoktm created this task.
Restricted Application added subscribers: Liuxinyu970226, Aklapper. · View Herald TranscriptJan 25 2020, 4:37 AM
Legoktm updated the task description. (Show Details)Jan 25 2020, 4:39 AM
thcipriani closed this task as Resolved.Jan 25 2020, 5:04 AM
thcipriani claimed this task.

Here's a threaddump I managed to grab before restarting: https://fastthread.io/my-thread-report.jsp?p=c2hhcmVkLzIwMjAvMDEvMjUvLS1hcGktOTY2ZjljZTItOGFkZC00NjdiLTg1ZDQtZjQ2MTIyNjNkNzYyODliNjM4MjctOTVjMS00YTlkLWIyZDMtOGMwMWQ5MTU0YTg0LnR4dC0t&

Looks like there was a blocked thread, but I'm unclear if that was due to memory issues that were already present.

Also there was a recent load/CPU spike -- this looks like Friday git gc activity

Most likely this machine needs some tuning for its specific traffic patterns, nothing looks out of the ordinary, really, just a JVM that couldn't handle the traffic. Filed as: T243655: Tune gerrit-replica to be more resilient to the kind of traffic it gets