Page MenuHomePhabricator

Update Jenkins hosts from Java 17 to Java 21
Open, In Progress, HighPublic

Description

I couldn't find a task for this yet, please close as dup if that wasn't the case.

Jenkins EOL for Java 17 is March 31st:

Java 17 end of life in Jenkins
You are running Jenkins on Java 17, support for which will end on or after Mar 31, 2026. Refer to the documentation for more details.

Upgrade guide here: https://www.jenkins.io/doc/book/platform-information/upgrade-java-to-21

The releases Jenkins are running bookworm and they can install Java 21, unfortunately the CI instances are still on bullseye, which will be a problem

Related Objects

Event Timeline

Change #1242483 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] releases: upgrade Java version from 17 to 21

https://gerrit.wikimedia.org/r/1242483

Change #1242483 merged by Dzahn:

[operations/puppet@production] releases: upgrade Java version from 17 to 21

https://gerrit.wikimedia.org/r/1242483

The plan is now to setup 2 (physical) machines and migrate either zuul or jenkins to them so that one can be upgraded without the other.

Change #1245280 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Enable Java 21 on build2002

https://gerrit.wikimedia.org/r/1245280

Change #1245280 merged by Muehlenhoff:

[operations/puppet@production] Enable Java 21 on build2002

https://gerrit.wikimedia.org/r/1245280

@Dzahn : I enabled Java 21 on build2002 (which is a Bookworm host) by merging https://gerrit.wikimedia.org/r/c/operations/puppet/+/1245280 and it worked just fine:

jmm@build2002:~$ sudo puppet agent -tv
Info: Using environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Info: Caching catalog for build2002.codfw.wmnet
Info: Applying configuration version '(69563b38df) Muehlenhoff - Enable Java 21 on build2002'
Notice: /Stage[main]/Java/Java::Package[openjdk-jdk-21]/Apt::Package_from_component[openjdk-21-jdk]/Apt::Repository[component-jdk21-apt.wikimedia.org-wikimedia-bookworm-wikimedia]/Concat[/etc/apt/sources.list.d/component-jdk21-apt.wiki\
media.org-wikimedia-bookworm-wikimedia.sources]/File[/etc/apt/sources.list.d/component-jdk21-apt.wikimedia.org-wikimedia-bookworm-wikimedia.sources]/ensure: defined content as '{sha256}18565268bcbfc08cad187c5c29e38a9271608b269e59d6934d\
ebf84620be9f3f'
Info: Concat[/etc/apt/sources.list.d/component-jdk21-apt.wikimedia.org-wikimedia-bookworm-wikimedia.sources]: Scheduling refresh of Exec[apt_repository_component-jdk21-apt.wikimedia.org-wikimedia-bookworm-wikimedia]
Notice: /Stage[main]/Java/Java::Package[openjdk-jdk-21]/Apt::Package_from_component[openjdk-21-jdk]/Apt::Repository[component-jdk21-apt.wikimedia.org-wikimedia-bookworm-wikimedia]/Exec[apt_repository_component-jdk21-apt.wikimedia.org-w\
ikimedia-bookworm-wikimedia]: Triggered 'refresh' from 1 event
Info: Apt::Repository[component-jdk21-apt.wikimedia.org-wikimedia-bookworm-wikimedia]: Scheduling refresh of Exec[apt_package_from_component_openjdk-21-jdk]
Notice: /Stage[main]/Java/Java::Package[openjdk-jdk-21]/Apt::Package_from_component[openjdk-21-jdk]/Exec[apt_package_from_component_openjdk-21-jdk]: Triggered 'refresh' from 1 event
Notice: /Stage[main]/Java/Java::Package[openjdk-jdk-21]/Apt::Package_from_component[openjdk-21-jdk]/Package[openjdk-21-jdk]/ensure: created
Notice: Applied catalog in 62.21 seconds
jmm@build2002:~$ /usr/lib/jvm/java-21-openjdk-amd64/bin/java -version
openjdk version "21.0.10" 2026-01-20
OpenJDK Runtime Environment (build 21.0.10+7-Debian-1deb12u1)
OpenJDK 64-Bit Server VM (build 21.0.10+7-Debian-1deb12u1, mixed mode, sharing)

I have validated the Jenkins Gearman plugin does work with Java 21 from Trixie (T420178). One less blocker!

One of the issue I flagged was whether a controller using Java 21 could communicate with agents running Java 17. They can't :-\

From https://www.jenkins.io/doc/book/platform-information/support-policy-java/

These requirements apply to all components of the Jenkins system, including the Jenkins controller, all types of agents, CLI clients, and other components.

That means:

  • production agents would need to be switched to the new hosts (contint1003 & contint2003). They would thus need the Puppet profiles profile::ci::docker, profile::ci::pipeline::publisher
  • WMCS agents are on Bullseye for which we do not have Java 17. We can either:
    1. rebuild the fleet with Bookworm with Java 17, and during the migration switch to our backported version of Java 21
    2. or build a new fleet of instances based on Trixie (which has Java 21) and only connect the new controller to it.

Rebuild them with Bookworm is filed as T421114. Else, if WMCS has enough space to allocate, the 2nd scenario is certainly easier migration wise, I have filed it as T421139.

Change #1260659 had a related patch set uploaded (by Hashar; author: Hashar):

[operations/puppet@production] ci: use docker.io package starting with Bookworm

https://gerrit.wikimedia.org/r/1260659

2.or build a new fleet of instances based on Trixie (which has Java 21) and only connect the new controller to it.

After thinking about it, I have abandoned the scenario to build a brand new fleet of Trixie instance and switch the service entirely to those. Instead I will rebuild them to Bookworm (T421114), this way we can start adding the instances to the old Jenkins that runs on Java 17 and validates everything is working. When doing the Java upgrade for the controller, we will need a Puppet patch to add and enable Java 21 on the Bookworm instances.

@Dzahn : I enabled Java 21 on build2002 (which is a Bookworm host) by merging https://gerrit.wikimedia.org/r/c/operations/puppet/+/1245280 and it worked just fine:

@MoritzMuehlenhoff sorry for the late reply. I just tested this again on releases1003 to see if I can repeat the issue.

root@releases1003:/# apt-get update
E: Conflicting values set for option Signed-By regarding source http://apt.wikimedia.org/wikimedia/ bookworm-wikimedia: /etc/apt/keyrings/wikimedia-archive-keyring.gpg != 
E: The list of sources could not be read.
Notice: /Stage[main]/Java/Java::Package[openjdk-jdk-21]/Apt::Package_from_component[openjdk-21-jdk]/Apt::Repository[component-jdk21-apt.wikimedia.org-wikimedia-bookworm-wikimedia]/Concat[/etc/apt/sources.list.d/component-jdk21-apt.wikimedia.org-wikimedia-bookworm-wikimedia.sources]/File[/etc/apt/sources.list.d/component-jdk21-apt.wikimedia.org-wikimedia-bookworm-wikimedia.sources]/ensure: defined content as '{sha256}18565268bcbfc08cad187c5c29e38a9271608b269e59d6934debf84620be9f3f'
Info: Concat[/etc/apt/sources.list.d/component-jdk21-apt.wikimedia.org-wikimedia-bookworm-wikimedia.sources]: Scheduling refresh of Exec[apt_repository_component-jdk21-apt.wikimedia.org-wikimedia-bookworm-wikimedia]
Notice: /Stage[main]/Java/Java::Package[openjdk-jdk-21]/Apt::Package_from_component[openjdk-21-jdk]/Apt::Repository[component-jdk21-apt.wikimedia.org-wikimedia-bookworm-wikimedia]/Exec[apt_repository_component-jdk21-apt.wikimedia.org-wikimedia-bookworm-wikimedia]/returns: E: Conflicting values set for option Signed-By regarding source http://apt.wikimedia.org/wikimedia/ bookworm-wikimedia: /etc/apt/keyrings/wikimedia-archive-keyring.gpg != 
Notice: /Stage[main]/Java/Java::Package[openjdk-jdk-21]/Apt::Package_from_component[openjdk-21-jdk]/Apt::Repository[component-jdk21-apt.wikimedia.org-wikimedia-bookworm-wikimedia]/Exec[apt_repository_component-jdk21-apt.wikimedia.org-wikimedia-bookworm-wikimedia]/returns: E: The list of sources could not be read.
Error: /Stage[main]/Java/Java::Package[openjdk-jdk-21]/Apt::Package_from_component[openjdk-21-jdk]/Apt::Repository[component-jdk21-apt.wikimedia.org-wikimedia-bookworm-wikimedia]/Exec[apt_repository_component-jdk21-apt.wikimedia.org-wikimedia-bookworm-wikimedia]: Failed to call refresh: '/usr/bin/apt-get update ' returned 100 instead of one of [0]
Error: /Stage[main]/Java/Java::Package[openjdk-jdk-21]/Apt::Package_from_component[openjdk-21-jdk]/Apt::Repository[component-jdk21-apt.wikimedia.org-wikimedia-bookworm-wikimedia]/Exec[apt_repository_component-jdk21-apt.wikimedia.org-wikimedia-bookworm-wikimedia]: '/usr/bin/apt-get update ' returned 100 instead of one of [0]
Info: Apt::Repository[component-jdk21-apt.wikimedia.org-wikimedia-bookworm-wikimedia]: Unscheduling all events on Apt::Repository[component-jdk21-apt.wikimedia.org-wikimedia-bookworm-wikimedia]
Notice: /Stage[main]/Java/Java::Package[openjdk-jdk-21]/Apt::Package_from_component[openjdk-21-jdk]/Exec[apt_package_from_component_openjdk-21-jdk]: Dependency Exec[apt_repository_component-jdk21-apt.wikimedia.org-wikimedia-bookworm-wikimedia] has failures: true
Warning: /Stage[main]/Java/Java::Package[openjdk-jdk-21]/Apt::Package_from_component[openjdk-21-jdk]/Exec[apt_package_from_component_openjdk-21-jdk]: Skipping because of failed dependencies
Warning: /Stage[main]/Java/Java::Package[openjdk-jdk-21]/Apt::Package_from_component[openjdk-21-jdk]/Package[openjdk-21-jdk]: Skipping because of failed dependencies
Warning: /Stage[main]/Java/Alternatives::Java[21]/Exec[update_java_alternatives_21]: Skipping because of failed dependencies
...
...

debugging now where it comes from.

root@releases1003:/etc/apt/sources.list.d# cat component-jdk21-apt.wikimedia.org-wikimedia-bookworm-wikimedia.sources 
# SPDX-License-Identifier: Apache-2.0
#
# This file is managed by puppet.
# Any local changes will be swiftly overwritten
#
# Most cloud-vps projects can make persistent changes to apt sources
# by adding a new .list file in /etc/apt/sources.list.d.
#
# Some cloud-vps projects have 'cloud.yaml:profile::apt::purge_sources'
# set to 'true', in which case apt sources can only be managed
# via puppet.
#
Types: deb deb-src
URIs: http://apt.wikimedia.org/wikimedia
Suites: bookworm-wikimedia
Components: component/jdk21
Signed-By: /etc/apt/keyrings/wikimedia-archive-keyring.gpg
 cat thirdparty-jenkins.sources 
# SPDX-License-Identifier: Apache-2.0
#
# This file is managed by puppet ...
Types: deb deb-src
URIs: http://apt.wikimedia.org/wikimedia
Suites: bookworm-wikimedia
Components: thirdparty/jenkins

^ Adding Signed-By: /etc/apt/keyrings/wikimedia-archive-keyring.gpg to the thirdparty-jenkins source config fixes the apt-get update.

Except puppet removes it again on next run.

found in modules/profile/manifests/bigtop/apt.pp which seems the only other place using a apt::repository { 'thirdparty-.

# Starting with Bookworm the Debian installer defaults to using the signed-by
# notation in apt-setup, also apply the same for the puppetised Wikimedia
# repository.
# The signed-by notation allows to specify which repository key is used
# for which repository (previously they applied to all repos)
# https://wiki.debian.org/DebianRepository/UseThirdParty
if debian::codename::ge('bookworm'){
    $wikimedia_apt_keyfile = 'puppet:///modules/install_server/autoinstall/keyring/wikimedia-archive-keyring.gpg'
} else {
    $wikimedia_apt_keyfile = undef
}

Change #1260766 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] ci: Add 'Signed-by' keyfile reference to thirdparty APT repo config

https://gerrit.wikimedia.org/r/1260766

Dzahn changed the task status from Open to In Progress.Wed, Mar 25, 6:26 PM
Dzahn triaged this task as High priority.

Mentioned in SAL (#wikimedia-operations) [2026-03-28T14:16:46Z] <mutante> releases1003 - re-enabled puppet which was disabled due to T418109 but should not have been disabled during switch of the deployment server; leading to T421532

Change #1260659 merged by Dzahn:

[operations/puppet@production] ci: use docker.io package starting with Bookworm

https://gerrit.wikimedia.org/r/1260659

I have deployed https://gerrit.wikimedia.org/r/c/operations/puppet/+/1260659

I confirmed it was noop on contint prod hosts. But it does show that on new integration-agent-docker-1070 listed above it will now install docker.io.

https://puppet-compiler.wmflabs.org/output/1260659/8372/

Change #1260766 merged by Dzahn:

[operations/puppet@production] ci: Add 'Signed-by' keyfile reference to thirdparty APT repo config

https://gerrit.wikimedia.org/r/1260766

Mentioned in SAL (#wikimedia-operations) [2026-04-02T19:09:11Z] <dzahn@cumin2002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on releases2003.codfw.wmnet with reason: T418109

Change #1267173 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] jenkins: add profile::ci::docker to role

https://gerrit.wikimedia.org/r/1267173

Change #1267290 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] ci: add missing .gpg extension to key file name

https://gerrit.wikimedia.org/r/1267290

Change #1267290 merged by Dzahn:

[operations/puppet@production] ci: add missing .gpg extension to key file name

https://gerrit.wikimedia.org/r/1267290

After fixing the APT key issue with:

https://gerrit.wikimedia.org/r/c/operations/puppet/+/1260766

https://gerrit.wikimedia.org/r/c/operations/puppet/+/1267290

I could then re-re-revert the change to switch Java version to 21 on releases hosts.

https://gerrit.wikimedia.org/r/c/operations/puppet/+/1267301

This installed Java 21 but did not remove Java 17.

[releases1003:~] $ dpkg -l | grep jre
ii  openjdk-17-jre:amd64                 17.0.18+8-1~deb12u1                  amd64        OpenJDK Java runtime, using Hotspot JIT
ii  openjdk-17-jre-headless:amd64        17.0.18+8-1~deb12u1                  amd64        OpenJDK Java runtime, using Hotspot JIT (headless)
ii  openjdk-21-jre:amd64                 21.0.10+7-1~deb12u1                  amd64        OpenJDK Java runtime, using Hotspot JIT
ii  openjdk-21-jre-headless:amd64        21.0.10+7-1~deb12u1                  amd64        OpenJDK Java runtime, using Hotspot JIT (headless)

sudo update-alternatives --config java and java -version shows that 21 is used as the default alternative.

So releases hosts are now done.

Change #1267173 merged by Dzahn:

[operations/puppet@production] jenkins: add profile::ci::docker to role

https://gerrit.wikimedia.org/r/1267173

Change #1268258 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] jenkins: switch firewall provider to ferm

https://gerrit.wikimedia.org/r/1268258

Change #1268258 merged by Dzahn:

[operations/puppet@production] jenkins: switch firewall provider to ferm

https://gerrit.wikimedia.org/r/1268258

Change #1268262 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] ci::docker: also install docker-cli when installing docker.io

https://gerrit.wikimedia.org/r/1268262

Change #1268262 merged by Dzahn:

[operations/puppet@production] ci::docker: also install docker-cli when installing docker.io

https://gerrit.wikimedia.org/r/1268262

The last couple changes fixed "puppet change on every run" and "failed docker-system-prune-all.service" by installing the docker-cli package along with the docker.io package.

Nowadays the client is not actually installed by the main package that installs the daemon anymore.

Change #1271017 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] jenkins: allow disabling jenkins even on the manager host

https://gerrit.wikimedia.org/r/1271017

Change #1271032 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] integration: switch integration-agent-docker VMs to Java 21

https://gerrit.wikimedia.org/r/1271032

Change #1271017 merged by Dzahn:

[operations/puppet@production] jenkins: allow disabling jenkins even on the manager host

https://gerrit.wikimedia.org/r/1271017

Change #1273919 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] contint: disable jenkins on legacy CI hosts

https://gerrit.wikimedia.org/r/1273919

Change #1274067 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] ci::docker: do not try to install docker-cli on bookworm

https://gerrit.wikimedia.org/r/1274067

Change #1274067 merged by Dzahn:

[operations/puppet@production] ci::docker: only install docker-cli if on trixie or newer

https://gerrit.wikimedia.org/r/1274067

Change #1275545 had a related patch set uploaded (by Dduvall; author: Dduvall):

[integration/pipelinelib@master] systemtests: Update jenkins to 2.555.1

https://gerrit.wikimedia.org/r/1275545

Change #1275545 merged by jenkins-bot:

[integration/pipelinelib@master] systemtests: Update jenkins to 2.555.1

https://gerrit.wikimedia.org/r/1275545