Page MenuHomePhabricator

Upgrade Cassandra clusters to Java 17
Open, MediumPublic

Description

In preparation for an upgrade to Debian Bookworm, upgrade the Cassandra clusters to Java 17.


There is no default configuration specific to Java 17 in Cassandra 4.1.8 (the version in production use at the time of writing), so Cassandra 5.0.6 has been used as starting point (where support is considered experimental).

modules/cassandra/templates/jvm17-server.options-4.x.erb (as compared to modules/cassandra/templates/jvm11-server.options-4.x.erb):

-XX:+ParallelRefProcEnabledWas unset in Java 11 config (defaults to enabled?). Explicitly enables parallel reference processing.
-XX:MaxTenuringThreshold=2Was unset/commented out for Java 11 (commented entry was '1'), now 2
-XX:G1HeapRegionSizeFrom 8m to 16m. We increased this to 8m in https://w.wiki/HvWr.
-XX:+UnlockExperimentalVMOptionsWas unset for Java 11, now (explicitly) enabled.
-XX:G0NewSizePercent=50Was unset in Java 11, now (explicitly) 50%.
-XX:MaxGCPauseMillis=300From unset (200ms?) to 300ms.
-XX:InitiatingHeapOccupancyPercent=70From unset (40%?) to 70%
-XX:ParallelGCThreads=16From unset in Java 11 to 16 threads now. Defaults to 5/8 the number of logical cores(?)

Event Timeline

Eevans triaged this task as Medium priority.
Eevans moved this task from Backlog to In-Progress on the User-Eevans board.

Change #1241042 had a related patch set uploaded (by Eevans; author: Eevans):

[operations/puppet@production] cassandra: enable use of Java 17

https://gerrit.wikimedia.org/r/1241042

Change #1242435 had a related patch set uploaded (by Eevans; author: Eevans):

[operations/puppet@production] cassandra-dev2001: enable Java 17

https://gerrit.wikimedia.org/r/1242435

Change #1241042 merged by Eevans:

[operations/puppet@production] cassandra: enable use of Java 17

https://gerrit.wikimedia.org/r/1241042

Change #1242435 merged by Eevans:

[operations/puppet@production] cassandra-dev2001: enable Java 17

https://gerrit.wikimedia.org/r/1242435

Change #1242447 had a related patch set uploaded (by Eevans; author: Eevans):

[operations/puppet@production] cassandra.in.sh: port Java 17'isms from 5.0 branch

https://gerrit.wikimedia.org/r/1242447

Change #1242447 merged by Eevans:

[operations/puppet@production] cassandra.in.sh: port Java 17'isms from 5.0 branch

https://gerrit.wikimedia.org/r/1242447

Change #1242461 had a related patch set uploaded (by Eevans; author: Eevans):

[operations/puppet@production] cassandra: argument typo

https://gerrit.wikimedia.org/r/1242461

Change #1242461 merged by Eevans:

[operations/puppet@production] cassandra: argument typo

https://gerrit.wikimedia.org/r/1242461

Seeing the following with Java 17 at startup on cassandra-dev2001 (startup fails):

ERROR [ScheduledTasks:1] 2026-02-23 17:47:34,556 JVMStabilityInspector.java:68 - Exception in thread Thread[ScheduledTasks:1,5,ScheduledTasks]
java.lang.NoClassDefFoundError: Could not initialize class org.apache.cassandra.service.StorageService
        at org.apache.cassandra.locator.DynamicEndpointSnitch.updateScores(DynamicEndpointSnitch.java:275)
        at org.apache.cassandra.locator.DynamicEndpointSnitch$1.run(DynamicEndpointSnitch.java:92)
        at org.apache.cassandra.concurrent.ExecutionFailure$1.run(ExecutionFailure.java:133)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
        at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
        at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: java.lang.ExceptionInInitializerError: Exception java.lang.ExceptionInInitializerError [in thread "ScheduledTasks:1"]
        at org.apache.cassandra.streaming.StreamManager.<init>(StreamManager.java:259)
        at org.apache.cassandra.streaming.StreamManager.<clinit>(StreamManager.java:58)
        at org.apache.cassandra.service.StorageService.<init>(StorageService.java:492)
        at org.apache.cassandra.service.StorageService.<clinit>(StorageService.java:308)
        ... 10 common frames omitted

Change #1242473 had a related patch set uploaded (by Eevans; author: Eevans):

[operations/puppet@production] cassandra: Java 8 no longer supported

https://gerrit.wikimedia.org/r/1242473

Seeing the following with Java 17 at startup on cassandra-dev2001 (startup fails):

ERROR [ScheduledTasks:1] 2026-02-23 17:47:34,556 JVMStabilityInspector.java:68 - Exception in thread Thread[ScheduledTasks:1,5,ScheduledTasks]
java.lang.NoClassDefFoundError: Could not initialize class org.apache.cassandra.service.StorageService
        at org.apache.cassandra.locator.DynamicEndpointSnitch.updateScores(DynamicEndpointSnitch.java:275)
        at org.apache.cassandra.locator.DynamicEndpointSnitch$1.run(DynamicEndpointSnitch.java:92)
        at org.apache.cassandra.concurrent.ExecutionFailure$1.run(ExecutionFailure.java:133)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
        at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
        at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: java.lang.ExceptionInInitializerError: Exception java.lang.ExceptionInInitializerError [in thread "ScheduledTasks:1"]
        at org.apache.cassandra.streaming.StreamManager.<init>(StreamManager.java:259)
        at org.apache.cassandra.streaming.StreamManager.<clinit>(StreamManager.java:58)
        at org.apache.cassandra.service.StorageService.<init>(StorageService.java:492)
        at org.apache.cassandra.service.StorageService.<clinit>(StorageService.java:308)
        ... 10 common frames omitted

This occurs when there is an exception during evaluation of a static initializer, or the initializer for a static variable. In this case it is a static member of StreamManager that is an instance of StreamManager (it's a singleton), the exception is coming from its constructor:

255: public StreamManager()
256: {
257:     DurationSpec.LongNanosecondsBound duration = DatabaseDescriptor.getStreamingStateExpires();
258:     long sizeBytes = DatabaseDescriptor.getStreamingStateSize().toBytes();
259:     long numElements = sizeBytes / StreamingState.ELEMENT_SIZE;
260:     logger.info("Storing streaming state for {} or for {} elements", duration, numElements);
261:     states = CacheBuilder.newBuilder()
262:                          .expireAfterWrite(duration.quantity(), duration.unit())
263:                          .maximumSize(numElements)
264:                          .build();
265: }

Line 259 is just...math, division. Both of the dividend and divisor are computed, so maybe it's a divide-by-zero, or overflow as the result of some Java 17ism? No idea.

Next steps would be to add some exception handling and try again, but it begs the question of whether this is a path we should be going down. Presumably this would not be an issue on Cassandra 5.0, which we will need to upgrade to eventually.

Was pointed to: https://github.com/chali/cassandra/commit/f899758, a changeset from someone who ported a 4.x version of Cassandra to Java 21.

image.png (60×480 px, 5 KB)

This is Java 21, not 17, but I suspect a fair proportion of this is relevant.

Change #1242473 merged by Eevans:

[operations/puppet@production] cassandra: Java 8 no longer supported

https://gerrit.wikimedia.org/r/1242473