Page MenuHomePhabricator

Consider using default JVM G1GC settings in the RESTBase Cassandra cluster
Closed, ResolvedPublic

Description

Best practice is to leave the G1 garbage-collector to its default settings (unless a good reason can be found to do otherwise). In the RESTBase Cassandra cluster we ended up with several settings that were operationalized without much in the way of justification. The lone exception, our use of 8MB region sizes, which were proven to avoid allocation of so-called Humongous Objects, which the collector handles in an exceptional (and sub-optimal) manner.

Setting the collector back to its defaults (sans the region size) has been tested in the dev environment with positive results. We should do the same in production.

Event Timeline

Eevans triaged this task as Medium priority.Apr 12 2018, 8:57 PM

Let's start with a couple of canaries in restbase1010 and restbase2003 respectively.

Mentioned in SAL (#wikimedia-operations) [2018-04-12T21:03:21Z] <urandom> temporarily disabling puppet to make (ephemeral) change to GC settings, restbase1010 -- T192112

Mentioned in SAL (#wikimedia-operations) [2018-04-12T21:06:59Z] <urandom> restarting Cassandra, restbase1010 -- T192112

Mentioned in SAL (#wikimedia-operations) [2018-04-12T22:07:52Z] <urandom> restarting Cassandra, restbase2003 -- T192112

Let's start with a couple of canaries in restbase1010 and restbase2003 respectively.

Done.

Change 426152 had a related patch set uploaded (by Eevans; owner: Eevans):
[operations/puppet@production] cassandra: restore (most) G1GC settings to defaults

https://gerrit.wikimedia.org/r/426152