| JAllemandou | |
| Wed, Feb 4, 3:32 PM |
| F71914439: image.png | |
| Thu, Feb 12, 2:22 PM |
| F71913620: image.png | |
| Thu, Feb 12, 2:22 PM |
| F71698600: image.png | |
| Fri, Feb 6, 7:29 AM |
| F71691534: image.png | |
| Thu, Feb 5, 2:11 PM |
When investigating for DDoS mitigation using Turnilo/Druid, SREs have experienced slow downs. @elukey and @CDanis pointed out that historical nodes GC was spiking at that time.
We wish to investigate as in
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | JAllemandou | T416493 Investigate druid historical nodes GC spikes | |||
| Unknown Object (Task) |
I have run a few queries, and managed to get the GC spike somehow. It's interesting to see that heap-space usage doesn't change at all during the spikes, so it's likely that the system uses a lot of short-lived objects.
@JAllemandou just to be on the same page - did you notice a slowdown in Turnilo when the spike happened?
I have experiences a slow-down yes, and query response time is higher, even if there also are some other artifacts at play with this metrics.
@JAllemandou nice thanks! I just added a series of panels in the Druid's grafana dashboard showing the various JVM's memory pools usage, I didn't spot anything so far but it may be useful. I kind of expected the Young gen / Eden's memory pool to increase in size, but I didn't find it in the metrics. At this point the best info may only come from enabling the GC logs, to understand what's happening.
I have ran an investigation getting GC logs while running queries with high GC spikes, and nothing special comes out of it. It just happens that Historical needs more young-heap space. I have quickly reviewed the curernt config we have for the cluster and will suggest changes early next week.
Change #1238695 had a related patch set uploaded (by Joal; author: Joal):
[operations/puppet@production] Update druid config to try to fix GC spikes
Change #1238695 merged by Btullis:
[operations/puppet@production] Update druid config to try to fix GC spikes
Mentioned in SAL (#wikimedia-analytics) [2026-02-11T15:56:06Z] <btullis> Deploying https://gerrit.wikimedia.org/r/c/operations/puppet/+/1238695 for T416493 and roll-restarting druid-analytics
Change #1238768 had a related patch set uploaded (by Joal; author: Joal):
[operations/puppet@production] Update druid analytics config
Change #1238768 merged by Btullis:
[operations/puppet@production] Update druid analytics config
Change #1239027 had a related patch set uploaded (by Joal; author: Joal):
[operations/puppet@production] Update druid analytics memory configuration
Change #1239027 merged by Btullis:
[operations/puppet@production] Update druid analytics memory configuration
With latest changes to memory configuration, we have lowered significantly the GC time for regex queries.
Taking this query: https://w.wiki/HodB when run over 30 days (thank you @CDanis !) as an example.
The original GC times as captured by Chris:
Closing this as resolved, please re-open as needed!