With T334517: upgrade contint servers to bullseye done I guess this just leaves the https://releases-jenkins.wikimedia.org hosts (/^releases[12]003\.(codfw|eqiad)\./) to upgrade to bullseye.

Change #1064437 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] releases: upgrade Java JDK version from 11 to 17

https://gerrit.wikimedia.org/r/1064437

In T359795#10082563, @bd808 wrote:

With T334517: upgrade contint servers to bullseye done I guess this just leaves the https://releases-jenkins.wikimedia.org hosts (/^releases[12]003\.(codfw|eqiad)\./) to upgrade to bullseye.

releases1003 and releases2003 got upgraded to Bulllseye a year ago via T334435 and Debian Bullseye ships OpenJDK 17.

I have reviewed Daniel patch Gerrit 1064437 - upgrade Java JDK version from 11 to 17 there is some oddity which is that for the release Jenkins, the systemd unit is overridden from the Jenkins deployment repo ( https://gitlab.wikimedia.org/repos/releng/jenkins-deploy/-/blob/master/scap/templates/override.conf.j2 ). @jnuche should be able to follow up review and test the release Jenkins with Java 17 with the development environment we have.

For the CI Jenkins, we need to verify the various plugins do work with Java 17. I am specially concerned by the Gearman plugin and the Gearman protocol library it depends on ( https://gerrit.wikimedia.org/g/integration/gearman-java ). Last time I have discovered the 10+ years old code was aiming at Java 5 and ended up hitting a breaking change introduced in Java 11 (JDK-8200458 / T271683). I will eventually file task(s) for it.

jnuche opened https://gitlab.wikimedia.org/repos/releng/jenkins-deploy/-/merge_requests/70

Draft: use default system Java in Jenkins systemd override file

Change #1064437 merged by Dzahn:

[operations/puppet@production] releases: upgrade Java JDK version from 11 to 17

https://gerrit.wikimedia.org/r/1064437

Change #1068801 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] releases: drop java_home parameter for releases/mediawiki

https://gerrit.wikimedia.org/r/1068801

Change #1068801 merged by Dzahn:

[operations/puppet@production] releases: drop java_home parameter for releases/mediawiki

https://gerrit.wikimedia.org/r/1068801

update, per the comments and plan on gerrit:

1. the java_home dir parameter is not passed to releases::mediawiki anymore from puppet, uses the default value from jenkins but this should have no effect

2. openjdk-17-jdk is now installed on releases* hosts by puppet, openjdk-11 packages are also installed as before

3. you can now deploy the actually relevant java_home dir change in the jenkins deploy repo in gitlab

4. once everything seems fine we will remove the jdk-11 packages from releases hosts

5. ticket should be resolved

cheers

Both package sets installed. update-alternatives did run automatically and set 17 to the new default.

java_home is under control of the deploy repo though.

[releases1003:~] $ dpkg -l | grep jdk
ii  openjdk-11-jdk:amd64                 11.0.24+8-2~deb11u1            amd64        OpenJDK Development Kit (JDK)
ii  openjdk-11-jdk-headless:amd64        11.0.24+8-2~deb11u1            amd64        OpenJDK Development Kit (JDK) (headless)
ii  openjdk-11-jre:amd64                 11.0.24+8-2~deb11u1            amd64        OpenJDK Java runtime, using Hotspot JIT
ii  openjdk-11-jre-headless:amd64        11.0.24+8-2~deb11u1            amd64        OpenJDK Java runtime, using Hotspot JIT (headless)
ii  openjdk-17-jdk:amd64                 17.0.12+7-2~deb11u1            amd64        OpenJDK Development Kit (JDK)
ii  openjdk-17-jdk-headless:amd64        17.0.12+7-2~deb11u1            amd64        OpenJDK Development Kit (JDK) (headless)
ii  openjdk-17-jre:amd64                 17.0.12+7-2~deb11u1            amd64        OpenJDK Java runtime, using Hotspot JIT
ii  openjdk-17-jre-headless:amd64        17.0.12+7-2~deb11u1            amd64        OpenJDK Java runtime, using Hotspot JIT (headless)

[releases1003:~] $ sudo update-alternatives --config  java
There are 2 choices for the alternative java (providing /usr/bin/java).

  Selection    Path                                         Priority   Status
------------------------------------------------------------
* 0            /usr/lib/jvm/java-17-openjdk-amd64/bin/java   1711      auto mode
  1            /usr/lib/jvm/java-11-openjdk-amd64/bin/java   1111      manual mode
  2            /usr/lib/jvm/java-17-openjdk-amd64/bin/java   1711      manual mode

hashar merged https://gitlab.wikimedia.org/repos/releng/jenkins-deploy/-/merge_requests/70

use default system Java in Jenkins systemd override file

The two Jenkins instances running on releases1003.eqiad.wmnet and releases2003.codfw.wmnet have been updated and now use Java 17 🎉

We can remove the openjdk-11-jdk package from both machines.

Please note we still need to the get the Java update to the CI boxes before we can close this ticket.

Mentioned in SAL (#wikimedia-operations) [2024-08-30T17:44:25Z] <mutante> releases1003/2003 - sudo apt-get remove openjdk-11-* - Java 11 has been replaced by Java 17 - T359795

Thank you. I removed the JDK 11 packages manually from both hosts. Puppet is not managing them anymore.

sudo apt-get remove openjdk-11-*

The following packages will be REMOVED:
  openjdk-11-jdk openjdk-11-jdk-headless openjdk-11-jre openjdk-11-jre-headless

Change #1069325 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] contint: add java jdk-17 packages in addition to jdk-11

https://gerrit.wikimedia.org/r/1069325

Change #1069327 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] contint: switch java_home from jdk-11 to jdk-17

https://gerrit.wikimedia.org/r/1069327

Change #1069328 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] contint: remove jdk-11 packages

https://gerrit.wikimedia.org/r/1069328

Change #1069325 merged by Dzahn:

[operations/puppet@production] contint: add java jdk-17 packages in addition to jdk-11

https://gerrit.wikimedia.org/r/1069325

Mentioned in SAL (#wikimedia-operations) [2024-09-03T16:17:00Z] <mutante> contint - installed java jdk 17 packages - just installed, in parallel to existing jdk 11, no change to java_home / what is used by CI yet. T359795

contint* machines now have both JDK 17 and JDK 11 installed:

But 11 is still the default:

[contint2002:~] $ sudo update-alternatives --config  java
There are 2 choices for the alternative java (providing /usr/bin/java).

  Selection    Path                                         Priority   Status
------------------------------------------------------------
  0            /usr/lib/jvm/java-17-openjdk-amd64/bin/java   1711      auto mode
* 1            /usr/lib/jvm/java-11-openjdk-amd64/bin/java   1111      manual mode
  2            /usr/lib/jvm/java-17-openjdk-amd64/bin/java   1711      manual mode

I have tested the Gearman plugin which was the one causing trouble when we upgraded to Java 11. I haven't tested all the other plugins though so I don't have a high confidence the CI Jenkins will work under Java 17, a plugin might well fail for whatever reason.

That being said, I guess you can now switch JAVA_HOME to Java 17 which is https://gerrit.wikimedia.org/r/c/operations/puppet/+/1069327/ and then restart the CI Jenkins (sudo systemctl start jenkins). Then later the Java 11 packages can be removed (the Java 11 CI jobs use containers).

I remember what is missing! Before removing Java 11, we need to change the default Java to point to Java 17. That is what the production agents are using.

So the sequence is:

• install new packages (done)
• Change default java to java 17:
  • make it appear first in hieradata/role/common/ci.yaml profile::java::java_packages
  • Run Puppet on contint*
  • Disconnect/reconnect agents
• Switch java_home, run Puppet which would update the systemd override
• Restart Jenkins
• Once everything looks fine, remove the Java 11 packages.

Then, I don't think we need to split the removal in multiple small patches, that confused me earlier today and the current series of patches misses the step to switch the agent to Java 17, and I think that is an unnecessary step.

What about a single Puppet change which removes the Java 11 package and set JAVA_HOME? Once Puppet ran, restart Jenkins. The updated JAVA_HOME would have Jenkins running under Java 17 and since Java 11 is no more in profile::java::java_packages, Java 17 would be the default and used by the production agent. Once all is fine, purge the Java 11 packages.

I would prefer the switch to happen during European morning (typically after the 9-10am CEST backport window). CI has less activity and its less disruptive, I and Jaime will also be able to monitor Jenkins and live debug if something turns out to be troublesome :)

we need to change the default Java to point to Java 17.

This would be achieved by us running sudo update-alternatives --config java and interactively selecting the other version.

For some reason on releases hosts this happened automatically but on contint it didn't. Which can both be seen as advantage or disadvantage depending how we want to deploy.

But my idea was:

• install additional packages (done)
• merge the patch that switches java_home and also run the command above to make 17 the default
• reconnect agents
• restart jenkins
• test.. wait ...
• eventually remove java 11 packages

The reason to have this in separate patches was that you can always revert back to 11 if needed for a while.

[contint2002:~] $ sudo update-alternatives --config  java
There are 2 choices for the alternative java (providing /usr/bin/java).

  Selection    Path                                         Priority   Status
------------------------------------------------------------
  0            /usr/lib/jvm/java-17-openjdk-amd64/bin/java   1711      auto mode
* 1            /usr/lib/jvm/java-11-openjdk-amd64/bin/java   1111      manual mode
  2            /usr/lib/jvm/java-17-openjdk-amd64/bin/java   1711      manual mode

Press <enter> to keep the current choice[*], or type selection number:

prefer the switch to happen during European morning (typically after the 9-10am CEST backport window).

ACK, will talk in my team how we can make that happen best.

In T359795#10134992, @Dzahn wrote:

we need to change the default Java to point to Java 17.

This would be achieved by us running sudo update-alternatives --config java and interactively selecting the other version.

For some reason on releases hosts this happened automatically but on contint it didn't. Which can both be seen as advantage or disadvantage depending how we want to deploy.

That is managed by the Java puppet module and Puppet changed the alternative when https://gerrit.wikimedia.org/r/c/operations/puppet/+/1064437 got applied:

-profile::releases::mediawiki::jenkins_java_home: "/usr/lib/jvm/java-11-openjdk-amd64"
-
 profile::java::java_packages:
-  - version: "11"
+  - version: "17"
     variant: "jdk"

The first entry in profile::java::java_packages crafts the update alternatives. That is a step I am adding in my comment above T359795#10134776:

Change default java to java 17:
make it appear first in hieradata/role/common/ci.yaml profile::java::java_packages

The reason to have this in separate patches was that you can always revert back to 11 if needed for a while.

Well it is still possible with a single patch :) That is what was done for the releases Jenkins (see above), but that is an implementation detail. You can amend your series by adding one that move Java 17 above in the list and Puppet will do the update-alternatives (else it will revert your manual change back to Java 11..).

Change #1069327 merged by EoghanGaffney:

[operations/puppet@production] contint: switch Jenkins to Java 17

https://gerrit.wikimedia.org/r/1069327

Mentioned in SAL (#wikimedia-operations) [2024-09-19T08:45:32Z] <hashar> Restarting CI Jenkins with Java 17 # T359795

In T359795#10148316, @hashar wrote:

The first entry in profile::java::java_packages crafts the update alternatives. That is a step I am adding in my comment above T359795#10134776:

Thanks! The part I didn't realize or expect here is that the order in Hiera mattered. Gotcha and thanks to Eoghan for amending my patch accordingly. :)

Well it is still possible with a single patch :) That is what was done for the releases Jenkins (see above), but that is an implementation detail. You can amend your series by adding one that move Java 17 above in the list

ACK, same as above applies. Glad it's sorted out now.

Change #1069328 merged by EoghanGaffney:

[operations/puppet@production] contint: remove jdk-11 packages

https://gerrit.wikimedia.org/r/1069328

Mentioned in SAL (#wikimedia-operations) [2024-11-05T15:18:09Z] <hashar> Switched WMCS integration instances from Java 11 to Java 17 via Horizon project wide config. That was forgotten in T359795 and blocks today Jenkins upgrade ( T379059 )

Mentioned in SAL (#wikimedia-operations) [2024-11-05T15:27:54Z] <hashar> Switched deployment-deploy04.deployment-prep.eqiad1.wikimedia.cloud to Java 17 # T359795

Mentioned in SAL (#wikimedia-operations) [2024-11-05T15:32:36Z] <hashar> Switched PCC workers to Java 17 via https://horizon.wikimedia.org/project/prefixpuppet/?tab=prefix_puppet__puppet-pcc-worker # T359795

I have updated all the WMCS instances to use Java 17 by setting via Horizon:

profile::java::java_packages:
- variant: jre-headless
  version: '17'

And running Puppet on them.