Page MenuHomePhabricator

Migrate Cassandra to Java 11
Closed, ResolvedPublic

Description

For Buster and Bullseye Java 8 is being provided in a component/jdk8 for applications relying on Java 8 (the default JRE in Bullseye is Java 11).

Since we have upgraded to Cassandra 4 by now (which supports Java 11), we should also move to Java 11.

Since Cassandra uses profile::java, this should be as simple as changing profile::java::java_packages in Hiera on a per-cluster basis along with service restarts.

  • aqs
  • cassandra-dev
  • ml-cache
  • restbase
  • sessionstore
Also requires removing openjdk-8 packages (ala dpkg --remove openjdk-8-jdk openjdk-8-jre openjdk-8-jdk-headless openjdk-8-jre-headless)

Event Timeline

Eevans triaged this task as Medium priority.Feb 22 2024, 5:44 PM

Change #1039813 had a related patch set uploaded (by Eevans; author: Eevans):

[operations/puppet@production] cassandra-dev2001: upgrade to Java 11

https://gerrit.wikimedia.org/r/1039813

Change #1039813 merged by Eevans:

[operations/puppet@production] cassandra-dev2001: upgrade to Java 11

https://gerrit.wikimedia.org/r/1039813

cassandra-dev2001-{a,b} have been upgraded to Java 11 (canaries).

Change #1040188 had a related patch set uploaded (by Eevans; author: Eevans):

[operations/puppet@production] cassandra-dev: upgrade to Java 11

https://gerrit.wikimedia.org/r/1040188

Change #1040188 merged by Eevans:

[operations/puppet@production] cassandra-dev: upgrade to Java 11

https://gerrit.wikimedia.org/r/1040188

cassandra-dev2001-{a,b} have been upgraded to Java 11 (canaries).

And now all of cassandra-dev.

Change #1041138 had a related patch set uploaded (by Eevans; author: Eevans):

[operations/puppet@production] aqs: Upgrade aqs1010 to Java 11 (canary)

https://gerrit.wikimedia.org/r/1041138

Change #1041138 merged by Eevans:

[operations/puppet@production] aqs: Upgrade aqs1010 to Java 11 (canary)

https://gerrit.wikimedia.org/r/1041138

Mentioned in SAL (#wikimedia-operations) [2024-06-10T14:47:56Z] <urandom> aqs1010: restarting cassandra to apply upgrade to Java 11 — T350567

24 hours(ish) on and the aqs1010-{a,b} upgrade to Java 11 is fairly anticlimactic. The only real difference I can see is in the heap utilization graph (and it's an improvement).

image.png (837×1 px, 573 KB)
aqs1010-a (java-11) vs aqs1016-b (java-8)
Eevans updated the task description. (Show Details)
Eevans updated the task description. (Show Details)

Change #1042234 had a related patch set uploaded (by Eevans; author: Eevans):

[operations/puppet@production] aqs: Upgrade cluster to Java 11

https://gerrit.wikimedia.org/r/1042234

Can you please uninstall openjdk-8-* on the migrated clusters? (simply run dpkg --remove openjdk-8-jdk openjre-8-jre openjdk-8-jdk-headless openjre-8-jre-headless via Cumin). This avoids confusion the next time we roll out new Java security updates.

Mentioned in SAL (#wikimedia-operations) [2024-06-12T13:18:24Z] <eevans@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on aqs1010.eqiad.wmnet with reason: Troubleshooting remote logging — T350567

Icinga downtime and Alertmanager silence (ID=36667e14-e2ab-458d-af49-c424df72a544) set by eevans@cumin1002 for 1 day, 0:00:00 on 1 host(s) and their services with reason: Troubleshooting remote logging — T350567

aqs1010.eqiad.wmnet

Mentioned in SAL (#wikimedia-operations) [2024-06-12T13:18:38Z] <eevans@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on aqs1010.eqiad.wmnet with reason: Troubleshooting remote logging — T350567

Change #1042273 had a related patch set uploaded (by Eevans; author: Eevans):

[operations/puppet@production] cassandra: alternate logging hostname definition

https://gerrit.wikimedia.org/r/1042273

Can you please uninstall openjdk-8-* on the migrated clusters? (simply run dpkg --remove openjdk-8-jdk openjre-8-jre openjdk-8-jdk-headless openjre-8-jre-headless via Cumin). This avoids confusion the next time we roll out new Java security updates.

Yup, will do!

Change #1042273 merged by Eevans:

[operations/puppet@production] cassandra: alternate logging hostname definition

https://gerrit.wikimedia.org/r/1042273

Change #1042234 merged by Eevans:

[operations/puppet@production] aqs: Upgrade cluster to Java 11

https://gerrit.wikimedia.org/r/1042234

Mentioned in SAL (#wikimedia-operations) [2024-06-13T19:28:32Z] <eevans@cumin1002> START - Cookbook sre.cassandra.roll-restart for nodes matching A:aqs-codfw: Upgrade to Java 11 — T350567 - eevans@cumin1002

Mentioned in SAL (#wikimedia-operations) [2024-06-13T20:55:47Z] <eevans@cumin1002> END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching A:aqs-codfw: Upgrade to Java 11 — T350567 - eevans@cumin1002

Mentioned in SAL (#wikimedia-operations) [2024-06-13T21:33:52Z] <eevans@cumin1002> START - Cookbook sre.cassandra.roll-restart for nodes matching A:aqs-eqiad: Upgrade to Java 11 — T350567 - eevans@cumin1002

Mentioned in SAL (#wikimedia-operations) [2024-06-13T23:02:38Z] <eevans@cumin1002> END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching A:aqs-eqiad: Upgrade to Java 11 — T350567 - eevans@cumin1002

Change #1046695 had a related patch set uploaded (by Eevans; author: Eevans):

[operations/puppet@production] restbase1028: Upgrade to Java 11

https://gerrit.wikimedia.org/r/1046695

Change #1046695 merged by Eevans:

[operations/puppet@production] restbase1028: Upgrade to Java 11

https://gerrit.wikimedia.org/r/1046695

Change #1047087 had a related patch set uploaded (by Eevans; author: Eevans):

[operations/puppet@production] restbase: upgrade cluster to Java 11

https://gerrit.wikimedia.org/r/1047087

Change #1047087 merged by Eevans:

[operations/puppet@production] restbase: upgrade cluster to Java 11

https://gerrit.wikimedia.org/r/1047087

Mentioned in SAL (#wikimedia-operations) [2024-06-18T16:22:58Z] <eevans@cumin1002> START - Cookbook sre.cassandra.roll-restart for nodes matching A:restbase-codfw: Upgrade to Java 11 — T350567 - eevans@cumin1002

Mentioned in SAL (#wikimedia-operations) [2024-06-18T18:29:42Z] <eevans@cumin1002> END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching A:restbase-codfw: Upgrade to Java 11 — T350567 - eevans@cumin1002

Mentioned in SAL (#wikimedia-operations) [2024-06-18T18:49:52Z] <eevans@cumin1002> START - Cookbook sre.cassandra.roll-restart for nodes matching A:restbase-eqiad: Upgrade to Java 11 — T350567 - eevans@cumin1002

Mentioned in SAL (#wikimedia-operations) [2024-06-18T20:59:53Z] <eevans@cumin1002> END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching A:restbase-eqiad: Upgrade to Java 11 — T350567 - eevans@cumin1002

Change #1047990 had a related patch set uploaded (by Eevans; author: Eevans):

[operations/puppet@production] sessionstore: Upgrade to Java 11

https://gerrit.wikimedia.org/r/1047990

Change #1047990 merged by Eevans:

[operations/puppet@production] sessionstore: Upgrade to Java 11

https://gerrit.wikimedia.org/r/1047990

Mentioned in SAL (#wikimedia-operations) [2024-06-20T14:22:40Z] <eevans@cumin1002> START - Cookbook sre.cassandra.roll-restart for nodes matching sessionstore2004.codfw.wmnet: Upgrade to Java 11 — T350567 - eevans@cumin1002

Mentioned in SAL (#wikimedia-operations) [2024-06-20T14:29:00Z] <eevans@cumin1002> END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching sessionstore2004.codfw.wmnet: Upgrade to Java 11 — T350567 - eevans@cumin1002

Mentioned in SAL (#wikimedia-operations) [2024-06-20T14:30:14Z] <eevans@cumin1002> START - Cookbook sre.cassandra.roll-restart for nodes matching sessionstore200[5-6].codfw.wmnet: Upgrade to Java 11 — T350567 - eevans@cumin1002

Mentioned in SAL (#wikimedia-operations) [2024-06-20T14:42:24Z] <eevans@cumin1002> END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching sessionstore200[5-6].codfw.wmnet: Upgrade to Java 11 — T350567 - eevans@cumin1002

Mentioned in SAL (#wikimedia-operations) [2024-06-20T14:46:14Z] <eevans@cumin1002> START - Cookbook sre.cassandra.roll-restart for nodes matching sessionstore100[4-6].eqiad.wmnet: Upgrade to Java 11 — T350567 - eevans@cumin1002

Mentioned in SAL (#wikimedia-operations) [2024-06-20T15:04:26Z] <eevans@cumin1002> END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching sessionstore100[4-6].eqiad.wmnet: Upgrade to Java 11 — T350567 - eevans@cumin1002

Eevans updated the task description. (Show Details)

Done!

macro-deployed