Page MenuHomePhabricator
Create Task
Maniphest T421114

Rebuild all Jenkins agents VM to Bookworm to support Java 21
Open, Needs TriagePublic
Actions

Description

To upgrade Java to version 21 on the Jenkins controller (T418109), we need need the same Java 21 on the agents. Jenkins Java Support Policy mentions:

These requirements apply to all components of the Jenkins system, including the Jenkins controller, all types of agents, CLI clients, and other components.

Our Jenkins agents are based on Bullseye which has Java 17 and are we not backporting Java 21 to Bullseye. We thus need to rebuild the instances.

We can rebuild the agents with either Bookworm or Trixie.

  • Trixie comes with Java 21 and does not have Java 17. That would prevent to attach the rebuild image to the controller which runs Java 17.
  • Bookworm has Java 17 and we have a backport of Java 21

Bookworm would let us rebuild the image with Java 17. Attach them to the Jenkins controller running Java 17. When upgrading the controller to Java 21, we can then switch the Bookworm instance to use Java 21 from component/jdk21.

List of agents on Bullseye

Agents already on Bookworm:

  • pcc-worker1004.puppet-diffs.eqiad1.wikimedia.cloud
  • pcc-worker1005.puppet-diffs.eqiad1.wikimedia.cloud
  • pcc-worker1006.puppet-diffs.eqiad1.wikimedia.cloud

Production hosts:

  • contint1002, contint2002 (they will be replaced by the new contint1003 / contint2003)

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
cloudbackup: do not back integration-castor* instancesoperations/puppetproduction+1 -1
jjb: use $CASTOR_HOST for Quibble success cacheintegration/configmaster+5 -5
ci::docker: also install docker-cli when installing docker.iooperations/puppetproduction+1 -1
ci: use docker.io package starting with Bookwormoperations/puppetproduction+27 -17
profile::ci::package_builder: create aptcache diroperations/puppetproduction+4 -1
profile::ci::package_builder: force link of pbuilder cacheoperations/puppetproduction+1 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

hashar created this task.Tue, Mar 24, 2:07 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptTue, Mar 24, 2:07 PM
hashar updated the task description. (Show Details)Tue, Mar 24, 2:12 PM
hashar added a parent task: T418109: Update Jenkins hosts from Java 17 to Java 21.
hashar mentioned this in T418109: Update Jenkins hosts from Java 17 to Java 21.Tue, Mar 24, 2:15 PM
hashar updated the task description. (Show Details)Tue, Mar 24, 2:28 PM
hashar mentioned this in T421139: Large temporary quota increase for integration project to ease Jenkins migration.Tue, Mar 24, 4:12 PM
hashar updated the task description. (Show Details)Tue, Mar 24, 5:50 PM
hashar renamed this task from Rebuild all Jenkins agents VM to support Java 21 to Rebuild all Jenkins agents VM to Bookworm to support Java 21.Wed, Mar 25, 11:20 AM
gerritbot added a comment.Wed, Mar 25, 11:56 AM

Change #1260659 had a related patch set uploaded (by Hashar; author: Hashar):

[operations/puppet@production] ci: use docker.io package starting with Bookworm

https://gerrit.wikimedia.org/r/1260659

gerritbot added a project: Patch-For-Review.Wed, Mar 25, 11:56 AM
hashar mentioned this in T421244: Replace deployment-deploy04 with a Bookworm instance with Java 21.Wed, Mar 25, 3:17 PM
hashar added a subtask: T421244: Replace deployment-deploy04 with a Bookworm instance with Java 21.
hashar updated the task description. (Show Details)Wed, Mar 25, 3:20 PM
hashar added a subtask: T421242: New flavor for the integration project with more vCPU and ephemeral disk space.Mon, Mar 30, 11:27 AM
taavi closed subtask T421242: New flavor for the integration project with more vCPU and ephemeral disk space as Resolved.Thu, Apr 2, 11:03 AM
hashar added a comment.Thu, Apr 2, 12:35 PM

From subtask T421242 we have a new flavor g4.cores10.ram24.disk20.ephemeral100.4xiops which has:

  • 2 more CPUs (8 → 10)
  • 10 more ephemeral disk space (90G → 100G)

I will look at rebuilding some instances this afternoon. I am not sure I will pool them immediately though.

hashar added a comment.Thu, Apr 2, 1:44 PM

I have created 7 instances with the g4.cores10.ram24.disk20.ephemeral100.4xiops flavor:

integration-agent-docker-1070
integration-agent-docker-1071
integration-agent-docker-1072
integration-agent-docker-1073
integration-agent-docker-1074
integration-agent-docker-1075
integration-agent-docker-1076

I have more or less followed the guide at https://wikitech.wikimedia.org/wiki/Nova_Resource:Integration/Setup .

Stashbot added a comment.Thu, Apr 2, 2:03 PM

Mentioned in SAL (#wikimedia-operations) [2026-04-02T14:03:07Z] <hashar> Jenkins CI: reloading configuration from disk to poll new nodes # T421114

gerritbot added a comment.Thu, Apr 2, 5:49 PM

Change #1260659 merged by Dzahn:

[operations/puppet@production] ci: use docker.io package starting with Bookworm

https://gerrit.wikimedia.org/r/1260659

Dzahn added a comment.Thu, Apr 2, 5:57 PM

I have deployed https://gerrit.wikimedia.org/r/c/operations/puppet/+/1260659

I confirmed it was noop on contint prod hosts. But it does show that on new integration-agent-docker-1070 listed above it will now install docker.io.

https://puppet-compiler.wmflabs.org/output/1260659/8372/

Maintenance_bot removed a project: Patch-For-Review.Thu, Apr 2, 6:31 PM
gerritbot added a comment.Thu, Apr 2, 7:12 PM

Change #1267173 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] jenkins: add profile::ci::docker to role

https://gerrit.wikimedia.org/r/1267173

gerritbot added a project: Patch-For-Review.Thu, Apr 2, 7:12 PM
Dzahn added a comment.Fri, Apr 3, 12:33 AM
In T421114#11782455, @hashar wrote:

I have created 7 instances with the g4.cores10.ram24.disk20.ephemeral100.4xiops flavor:

integration-agent-docker-1070
integration-agent-docker-1071
integration-agent-docker-1072
integration-agent-docker-1073
integration-agent-docker-1074
integration-agent-docker-1075
integration-agent-docker-1076

I have confirmed on all the machines above that docker.io is now installed.

hashar mentioned this in T422234: Remove the "blubber" label from Jenkins agents.Fri, Apr 3, 6:52 AM
hashar added a comment.Fri, Apr 3, 9:14 AM
In T421114#11783884, @Dzahn wrote:

I have deployed https://gerrit.wikimedia.org/r/c/operations/puppet/+/1260659

I confirmed it was noop on contint prod hosts. But it does show that on new integration-agent-docker-1070 listed above it will now install docker.io.

https://puppet-compiler.wmflabs.org/output/1260659/8372/

Indeed! Note: I already had the change cherry picked on the CI Puppetmaster to unblock the Puppet run on those hosts :)

Stashbot added a comment.Fri, Apr 3, 9:15 AM

Mentioned in SAL (#wikimedia-releng) [2026-04-03T09:15:10Z] <hashar> Added Bookworm based Jenkins agents to the pool with label Docker. Hostnames are integration-agent-docker-107* # T421114

hashar added a comment.Fri, Apr 3, 1:01 PM

I have created 6 more instances with the g4.cores10.ram24.disk20.ephemeral100.4xiops flavor:

integration-agent-docker-1077
integration-agent-docker-1078
integration-agent-docker-1079
integration-agent-docker-1080
integration-agent-docker-1081
integration-agent-docker-1082

hashar mentioned this in T422250: composer-package-php jobs failure with "Unable to find image 'docker-registry.wikimedia.org/bookworm:latest' locally".Fri, Apr 3, 2:09 PM
hashar added a subtask: T422250: composer-package-php jobs failure with "Unable to find image 'docker-registry.wikimedia.org/bookworm:latest' locally".
hashar closed subtask T422250: composer-package-php jobs failure with "Unable to find image 'docker-registry.wikimedia.org/bookworm:latest' locally" as Resolved.Fri, Apr 3, 2:18 PM
gerritbot added a comment.Fri, Apr 3, 3:23 PM

Change #1267883 had a related patch set uploaded (by Hashar; author: Hashar):

[operations/puppet@production] profile::ci::package_builder: force link of pbuilder cache

https://gerrit.wikimedia.org/r/1267883

hashar updated the task description. (Show Details)Fri, Apr 3, 3:32 PM
gerritbot added a comment.Fri, Apr 3, 3:37 PM

Change #1267887 had a related patch set uploaded (by Hashar; author: Hashar):

[operations/puppet@production] profile::ci::package_builder: create aptcache dir

https://gerrit.wikimedia.org/r/1267887

hashar added a subtask: T422258: Puppet fail to create volume group for ephemeral disk space when it is sda (instead of sdb).Fri, Apr 3, 3:48 PM
hashar updated the task description. (Show Details)Fri, Apr 3, 4:05 PM
hashar updated the task description. (Show Details)Fri, Apr 3, 4:11 PM
gerritbot added a comment.Fri, Apr 3, 4:58 PM

Change #1267883 merged by Dzahn:

[operations/puppet@production] profile::ci::package_builder: force link of pbuilder cache

https://gerrit.wikimedia.org/r/1267883

gerritbot added a comment.Fri, Apr 3, 4:58 PM

Change #1267887 merged by Dzahn:

[operations/puppet@production] profile::ci::package_builder: create aptcache dir

https://gerrit.wikimedia.org/r/1267887

hashar claimed this task.Mon, Apr 6, 7:56 AM
Peter subscribed.Mon, Apr 6, 5:11 PM
gerritbot added a comment.Mon, Apr 6, 6:20 PM

Change #1268262 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] ci::docker: also install docker-cli when installing docker.io

https://gerrit.wikimedia.org/r/1268262

gerritbot added a comment.Mon, Apr 6, 6:23 PM

Change #1268262 merged by Dzahn:

[operations/puppet@production] ci::docker: also install docker-cli when installing docker.io

https://gerrit.wikimedia.org/r/1268262

Stashbot added a comment.Tue, Apr 7, 7:54 AM

Mentioned in SAL (#wikimedia-operations) [2026-04-07T07:54:29Z] <hashar> Moved operations-puppet-tests-bullseye job from a Jenkins agent running Bullseye to one running Bookworm. The image is still on Bullseye! | T421114

Stashbot added a comment.Tue, Apr 7, 8:05 AM

Mentioned in SAL (#wikimedia-operations) [2026-04-07T08:05:01Z] <hashar> Moved Debian Glue jobs to Jenkins agents running Bookworm (integration-agent-pkgbuilder-1005 and integration-agent-pkgbuilder-1006)| T421114

hashar updated the task description. (Show Details)Tue, Apr 7, 8:09 AM
Stashbot added a comment.Tue, Apr 7, 10:09 AM

Mentioned in SAL (#wikimedia-releng) [2026-04-07T10:09:49Z] <hashar> Added Bookworm based Jenkins agents to the pool Hostnames 1083 to 1089 # T421114

hashar mentioned this in T420700: Evaluate if larger worker instances can increase the speed of the test runs.Tue, Apr 7, 10:14 AM
hashar updated the task description. (Show Details)Tue, Apr 7, 10:35 AM
Stashbot added a comment.Tue, Apr 7, 11:14 AM

Mentioned in SAL (#wikimedia-releng) [2026-04-07T11:14:34Z] <hashar> Added Bookworm based Jenkins agents to the pool Hostnames 1090, 1091, 1092 and 1093 # T421114

Stashbot added a comment.Tue, Apr 7, 12:17 PM

Mentioned in SAL (#wikimedia-releng) [2026-04-07T12:17:30Z] <hashar> Migrated Castor from integration-castor05 to integration-castor06. Updated CASTOR_HOST in Jenkins and moved the Cinder volume to the new instance # T421114

hashar updated the task description. (Show Details)Tue, Apr 7, 12:18 PM
gerritbot added a comment.Tue, Apr 7, 12:21 PM

Change #1268540 had a related patch set uploaded (by Hashar; author: Hashar):

[operations/puppet@production] cloudbackup: do not back integration-castor* instances

https://gerrit.wikimedia.org/r/1268540

gerritbot added a comment.Tue, Apr 7, 12:34 PM

Change #1268545 had a related patch set uploaded (by Hashar; author: Hashar):

[integration/config@master] jjb: use $CASTOR_HOST for Quibble success cache

https://gerrit.wikimedia.org/r/1268545

Stashbot added a comment.Tue, Apr 7, 12:36 PM

Mentioned in SAL (#wikimedia-releng) [2026-04-07T12:35:59Z] <hashar> jjb: use $CASTOR_HOST for Quibble success cache. https://gerrit.wikimedia.org/r/1268545 | This causes the Quibble jobs to use a new instance for the success cache, which is empty # T383243 T421114

Stashbot mentioned this in T383243: Zuul/Jenkins: Investigate caching of build results for MediaWiki testsuite jobs.Tue, Apr 7, 12:36 PM
gerritbot added a comment.Tue, Apr 7, 12:40 PM

Change #1268545 merged by jenkins-bot:

[integration/config@master] jjb: use $CASTOR_HOST for Quibble success cache

https://gerrit.wikimedia.org/r/1268545

gerritbot added a comment.Tue, Apr 7, 12:46 PM

Change #1268540 merged by David Caro:

[operations/puppet@production] cloudbackup: do not back integration-castor* instances

https://gerrit.wikimedia.org/r/1268540

hashar mentioned this in T422488: Rebuild integration-agent-qemu-1003 from Bullseye to Bookworm.Tue, Apr 7, 1:05 PM
hashar added a subtask: T422488: Rebuild integration-agent-qemu-1003 from Bullseye to Bookworm.
hashar updated the task description. (Show Details)
hashar closed subtask T422488: Rebuild integration-agent-qemu-1003 from Bullseye to Bookworm as Resolved.Wed, Apr 8, 6:26 AM
hashar updated the task description. (Show Details)
Stashbot added a comment.Wed, Apr 8, 6:31 AM

Mentioned in SAL (#wikimedia-releng) [2026-04-08T06:31:37Z] <hashar> Deleted integration-agent-castor05 Bullseye instance, replaced by integration-agent-castor06 which is on Bookworm # T421114

hashar updated the task description. (Show Details)Wed, Apr 8, 6:38 AM
Peter mentioned this in T420590: Decrease feedback loop for core tests to less than 10 minutes.Wed, Apr 8, 1:42 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits