Page MenuHomePhabricator

Enable GC (garbage collection) logs on Elasticsearch JVM
Closed, ResolvedPublic

Description

Recent issue on Elasticsearch indicates a GC overload. Collecting GC logs would help diagnose this kind of issue if it ever happens again. Some care needs to be taken around log rotation (GC logging is overly optimized and creates a few issues for log rotation).

Details

Related Gerrit Patches:

Event Timeline

Gehel created this task.May 10 2016, 9:12 AM
debt renamed this task from Enable GC logs on Elasticsearch JVM to Enable GC (garbage collection) logs on Elasticsearch JVM.Jun 16 2016, 10:05 PM
Dzahn removed a subscriber: Dzahn.Jun 21 2016, 3:42 AM
debt lowered the priority of this task from High to Medium.Jul 12 2016, 10:16 PM
debt edited projects, added Discovery-Search; removed Discovery-Search (Current work).
debt added a subscriber: debt.

moving to the backlog board until we have more time to look at this.

greg added a subscriber: greg.Sep 29 2016, 7:41 PM

This follow-up task from an incident report has not been updated recently. If it is no longer valid, please add a comment explaining why. If it is still valid, please prioritize it appropriately relative to your other work. If you have any questions, feel free to ask me (Greg Grossmeier).

debt added a comment.Sep 30 2016, 3:00 PM

Hi @Gehel and @EBernhardson - can we take a look at this to see if it's still valid?

@greg, the answer to your concerns might have to wait until early next week when @Gehel returns from his offsite travels, but thanks for bringing it up!

debt moved this task from needs triage to Up Next on the Discovery-Search board.Oct 5 2016, 7:47 PM

Let's go ahead and start working on this next.

debt assigned this task to Gehel.Oct 25 2016, 5:46 PM
debt edited projects, added Discovery-Search (Current work); removed Discovery-Search.

Change 318055 had a related patch set uploaded (by Gehel):
elasticsearch - enable garbage collection logs on relforge servers

https://gerrit.wikimedia.org/r/318055

Change 318055 merged by Gehel:
elasticsearch - enable garbage collection logs on relforge servers

https://gerrit.wikimedia.org/r/318055

Mentioned in SAL (#wikimedia-operations) [2016-10-27T12:04:25Z] <gehel> restart elasticsearch on relforge to activate GC logs - T134853

Puppet change is deployed. GC logs are available on relforge. I will wait a few days to check everything works fine before activating them on production clusters as well.

Change 318353 had a related patch set uploaded (by Gehel):
elasticsearch - enable GC logs by default

https://gerrit.wikimedia.org/r/318353

Deskana closed this task as Resolved.Oct 28 2016, 4:22 AM
Deskana added a subscriber: Deskana.

I will assume that @Gehel's patch resolves this issue. Please reopen if there are any issues. :-)

Gehel reopened this task as Open.Oct 31 2016, 4:15 PM

The patch *should* resolve the issue, but it is not yet deployed. So at this point GC logs are enabled on relforge cluster, but not anywhere else. I'm reopening this and will close it for real fairly soon.

Change 318353 merged by Gehel:
elasticsearch - enable GC logs by default

https://gerrit.wikimedia.org/r/318353

Gehel closed this task as Resolved.Nov 1 2016, 9:33 AM
Dzahn added a comment.EditedNov 1 2016, 7:12 PM

@Gehel Maybe we can share learnings about GC and which GC to use in particular over here where we are trying to optimize Gerrit in T148478 T148478#2750773 T148478#2760146 Are you using G1?

fwiw, here's where we enabled GC logging https://gerrit.wikimedia.org/r/#/c/317582/ https://gerrit.wikimedia.org/r/#/c/318067/