Elasticsearch is slow and spending lots of time Garbage Collecting
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Tarrow
	May 23 2022, 9:40 AM

Project Tags

Referenced Files

None

Subscribers

Aklapper

Tarrow

WMDE-leszek

Description

See: https://cloudlogging.app.goo.gl/4JomaL9ixawEpqLQ8

We see INFOs like this:

"[2022-05-23T08:41:26,534][INFO ][o.e.m.j.JvmGcMonitorService] [elasticsearch-master-0] [gc][old][239079][1736] duration [7.4s], collections [1]/[7.5s], total [7.4s]/[3.6h], memory [1.9gb]->[1.9gb]/[2gb], all_pools {[young] [2mb]->[2mb]/[0b]}{[old] [1.9gb]->[1.9gb]/[2gb]}{[survivor] [0b]->[0b]/[0b]}"
timestamp: "2022-05-23T08:41:26.535203976Z"

and WARNS like this:

"[2022-05-23T08:41:26,535][WARN ][o.e.m.j.JvmGcMonitorService] [elasticsearch-master-0] [gc][239079] overhead, spent [7.4s] collecting in the last [7.5s]"

Notably these seem to only be coming from elasticsearch nodes 0 and 1 at this time.

Related Objects

Mentioned In: T309019: 🟪️ [EPIC] Understand and solve (short term) Elasticsearch issues on Wikibase Cloud
T308483: 🟧 Perform migration of batch B

Event Timeline

Tarrow created this task.May 23 2022, 9:40 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 23 2022, 9:40 AM

In an attempt to resolve this issue we have increase the cpu limit for elasticsearch pods from 1250 to 1950: https://github.com/wmde/wbaas-deploy/pull/331

This change will take some time to apply since for ElasticSearch to start from cold it takes like 40mins+ and it needs to roll through all three nodes.

I expect to see these garbage log lines drop to 0 and only restart after we start making more indices.

Tarrow mentioned this in T308483: 🟧 Perform migration of batch B.May 23 2022, 9:47 AM

This seems to have eventually brought all nodes into a not-ready state. While ES 2 was being restarted with new CPU limits both 0 and 1 are now down. 1 seems to have crashed and restarted

Possibly we should consider using https://www.elastic.co/guide/en/elasticsearch/reference/6.8/indices-flush.html to reduce the startup time

WMDE-leszek subscribed.May 23 2022, 10:50 AM

WMDE-leszek mentioned this in T309019: 🟪️ [EPIC] Understand and solve (short term) Elasticsearch issues on Wikibase Cloud.May 23 2022, 10:53 AM

Looks like 0 and 1 both fell over after throwing an OOM error form the JVM. Now all of them are restarting

We have some metrics on the incidence of the GC errors but it would be nice to collect some on the *time* spent garbage collecting

After doubling the CPU and RAM (to 3.95cpu and 8gm RAM and 4GB Heap) yesterday.

This appears to be stable for today. We delete a non-master node and it came back up in under 30mins

Tarrow closed this task as Resolved.May 25 2022, 2:42 PM

Tarrow moved this task from Doing to Done on the Wikibase Cloud (Launch Migration Kanban (2022)) board.

Elasticsearch is slow and spending lots of time Garbage CollectingClosed, ResolvedPublicActions

Description

Related Objects

Event Timeline

Elasticsearch is slow and spending lots of time Garbage Collecting
Closed, ResolvedPublic
Actions