Page MenuHomePhabricator
Create Task
Maniphest T269354

Switch Jenkins servers to Java 11
Closed, ResolvedPublic
Actions

Description

When we have upgraded the Jenkins master from Debian Stretch to Debian Buster, we have kept using Java 8 notably because of the Gearman plugin on the CI ones. The releases one can be switched though.

The Jenkins agents are already running using Java 11.

A prerequisite for CI is to upgrade the Gearman plugin. There is a fork of the OpenDev plugin somewhere on Github and it will need to be forked on our Gerrit and re build.

  • releases1002.eqiad.wmnet
  • releases2001.codfw.wmnet
  • New Gearman plugin
    • contint2001.wikimedia.org
    • contint1001.wikimedia.org
  • Remove Java 8 from hosts

Details

SubjectRepoBranchLines +/-
contint: remove erroneous hiera setting for labsoperations/puppetproduction+0 -3
contint: use Java 11 on Jenkins agentsoperations/puppetproduction+8 -7
puppet_compiler: stop installing openjdkoperations/puppetproduction+1 -5
releases/ci: Remove now obsolete Java 8 packagesoperations/puppetproduction+0 -4
ci: switch Jenkins to Java 11operations/puppetproduction+2 -5
releases: switch Jenkins to Java 11operations/puppetproduction+2 -5
jenkins: support changing $JAVA_HOMEoperations/puppetproduction+28 -8
Customize query in gerrit

Related Objects
Search...

Event Timeline

hashar created this task.Dec 3 2020, 2:02 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptDec 3 2020, 2:02 PM
hashar added a project: Jenkins.Dec 3 2020, 2:02 PM
MoritzMuehlenhoff subscribed.Dec 3 2020, 2:04 PM
gerritbot added a comment.Dec 3 2020, 2:09 PM

Change 645075 had a related patch set uploaded (by Hashar; owner: Hashar):
[operations/puppet@production] jenkins: support changing $JAVA_HOME

https://gerrit.wikimedia.org/r/645075

gerritbot added a project: Patch-For-Review.Dec 3 2020, 2:09 PM
hashar mentioned this in T269352: Upgrade Jenkins to 2.263.1.Dec 3 2020, 2:12 PM
hashar edited projects, added Release-Engineering-Team (CI & Testing services); removed Release-Engineering-Team (Development services).Dec 8 2020, 1:04 PM
gerritbot added a comment.Dec 14 2020, 11:11 PM

Change 645075 merged by Dzahn:
[operations/puppet@production] jenkins: support changing $JAVA_HOME

https://gerrit.wikimedia.org/r/645075

Maintenance_bot removed a project: Patch-For-Review.Dec 15 2020, 12:10 AM
hashar claimed this task.Jan 11 2021, 8:54 AM
hashar added a project: Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)).
hashar moved this task from INBOX to Maintenance on the Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)) board.
hashar added a subtask: T271683: Upgrade Jenkins Gearman plugin from a forked repo.
hashar triaged this task as Medium priority.Feb 5 2021, 1:37 PM
hashar closed subtask T271683: Upgrade Jenkins Gearman plugin from a forked repo as Resolved.Feb 23 2021, 7:18 AM
hashar added a comment.Feb 23 2021, 7:23 AM

I have ported the Jenkins Gearman plugin to JDK 11, we can now upgrade the CI Jenkins.

MoritzMuehlenhoff added a comment.Feb 23 2021, 3:05 PM

Let's start with the releases Jenkins install? Given jenkins being managed by profile::java we only need to tweak the Hiera settings in hieradata/role/common/releases.yaml

gerritbot added a comment.Mar 11 2021, 8:13 AM

Change 670767 had a related patch set uploaded (by Hashar; owner: Hashar):
[operations/puppet@production] ci: switch Jenkins to Java 11

https://gerrit.wikimedia.org/r/670767

gerritbot added a project: Patch-For-Review.Mar 11 2021, 8:13 AM
gerritbot added a comment.Mar 11 2021, 8:17 AM

Change 670768 had a related patch set uploaded (by Hashar; owner: Hashar):
[operations/puppet@production] releases: switch Jenkins to Java 11

https://gerrit.wikimedia.org/r/670768

gerritbot added a comment.Mar 11 2021, 8:47 AM

Change 670768 merged by Muehlenhoff:
[operations/puppet@production] releases: switch Jenkins to Java 11

https://gerrit.wikimedia.org/r/670768

gerritbot added a comment.Mar 11 2021, 9:23 AM

Change 670767 merged by Muehlenhoff:
[operations/puppet@production] ci: switch Jenkins to Java 11

https://gerrit.wikimedia.org/r/670767

hashar added a comment.Mar 11 2021, 9:37 AM

@MoritzMuehlenhoff and I have conducted the Java upgrade for all Jenkins instances. Notably the CI one is working properly (after Gearman support got updated via T271683). I will keep monitoring it but it seems to behave properly.

Left to do will be to clean up the Java 8 package once we are sure everything works fine.

gerritbot added a comment.Mar 11 2021, 9:38 AM

Change 670776 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] releases/ci: Remove now obsolete Java 8 packages

https://gerrit.wikimedia.org/r/670776

hashar updated the task description. (Show Details)Mar 11 2021, 11:42 AM
gerritbot added a comment.Mar 16 2021, 8:35 AM

Change 670776 merged by Muehlenhoff:
[operations/puppet@production] releases/ci: Remove now obsolete Java 8 packages

https://gerrit.wikimedia.org/r/670776

Stashbot added a comment.Mar 16 2021, 8:42 AM

Mentioned in SAL (#wikimedia-operations) [2021-03-16T08:42:39Z] <moritzm> remove Java 8 from contint/releases T269354

MoritzMuehlenhoff updated the task description. (Show Details)Mar 16 2021, 8:43 AM
hashar added a comment.Mar 16 2021, 8:51 AM

Thank you @MoritzMuehlenhoff for the assistance in deploying the puppet patch and the overall java expertise.

I have checked the various Jenkins agents and some are still running Java 8.

We can check them with https://github.com/jenkinsci/jenkins-scripts/blob/master/scriptler/showAgentJavaVersion.groovy

import hudson.model.Node
import hudson.model.Slave
import jenkins.model.Jenkins

Jenkins jenkins = Jenkins.instance
for (Node node in jenkins.nodes) {
  // Make sure slave is online
  if (!node.toComputer().online) {
    println "Node '$node.nodeName' is currently offline - skipping check"
    continue;
  } else {
    props = node.toComputer().getSystemProperties();
    println "Node '$node.nodeName' is running " + props.get('java.runtime.version');
  }
}

To be pasted at:

releases (good)
Node 'releases1002.eqiad.wmnet' is running 11.0.9.1+1-post-Debian-1deb10u2
contint
Node 'compiler1001.puppet-diffs.eqiad.wmflabs' is running 11.0.9.1+1-post-Debian-1deb10u2
Node 'compiler1002.puppet-diffs.eqiad.wmflabs' is running 11.0.9.1+1-post-Debian-1deb10u2
Node 'compiler1003.puppet-diffs.eqiad.wmflabs' is running 11.0.9.1+1-post-Debian-1deb10u2
Node 'contint1001' is running 11.0.9.1+1-post-Debian-1deb10u2
Node 'contint2001' is running 11.0.9.1+1-post-Debian-1deb10u2
Node 'deployment-deploy01' is running 1.8.0_275-8u275-b01-1~deb9u1-b01
Node 'integration-agent-docker-1001' is running 1.8.0_275-8u275-b01-1~deb9u1-b01
Node 'integration-agent-docker-1002' is running 1.8.0_275-8u275-b01-1~deb9u1-b01
Node 'integration-agent-docker-1003' is running 1.8.0_275-8u275-b01-1~deb9u1-b01
Node 'integration-agent-docker-1004' is running 1.8.0_275-8u275-b01-1~deb9u1-b01
Node 'integration-agent-docker-1005' is running 1.8.0_275-8u275-b01-1~deb9u1-b01
Node 'integration-agent-docker-1006' is running 1.8.0_275-8u275-b01-1~deb9u1-b01
Node 'integration-agent-docker-1007' is running 1.8.0_275-8u275-b01-1~deb9u1-b01
Node 'integration-agent-docker-1008' is running 1.8.0_275-8u275-b01-1~deb9u1-b01
Node 'integration-agent-docker-1009' is running 1.8.0_275-8u275-b01-1~deb9u1-b01
Node 'integration-agent-docker-1010' is running 1.8.0_275-8u275-b01-1~deb9u1-b01
Node 'integration-agent-docker-1011' is running 1.8.0_275-8u275-b01-1~deb9u1-b01
Node 'integration-agent-docker-1012' is running 1.8.0_275-8u275-b01-1~deb9u1-b01
Node 'integration-agent-docker-1013' is running 1.8.0_275-8u275-b01-1~deb9u1-b01
Node 'integration-agent-docker-1014' is running 1.8.0_275-8u275-b01-1~deb9u1-b01
Node 'integration-agent-docker-1016' is running 1.8.0_275-8u275-b01-1~deb9u1-b01
Node 'integration-agent-docker-1020' is running 1.8.0_275-8u275-b01-1~deb9u1-b01
Node 'integration-agent-pkgbuilder-1001' is running 11.0.9.1+1-post-Debian-1deb10u2
Node 'integration-agent-pkgbuilder-1002' is running 11.0.9.1+1-post-Debian-1deb10u2
Node 'integration-agent-puppet-docker-1002' is running 1.8.0_275-8u275-b01-1~deb9u1-b01
Node 'integration-agent-qemu-1001' is running 1.8.0_275-8u275-b01-1~deb9u1-b01
Node 'integration-castor03' is running 1.8.0_275-8u275-b01-1~deb9u1-b01

Which is a bit unexpected since I am pretty sure I did the agents migration a while back :\

gerritbot added a comment.Mar 16 2021, 9:06 AM

Change 672658 had a related patch set uploaded (by Hashar; owner: Hashar):
[operations/puppet@production] contint: use Java 11 on Jenkins agents

https://gerrit.wikimedia.org/r/672658

gerritbot added a comment.Mar 16 2021, 9:23 AM

Change 672660 had a related patch set uploaded (by Hashar; owner: Hashar):
[operations/puppet@production] puppet_compiler: stop installing openjdk

https://gerrit.wikimedia.org/r/672660

gerritbot added a comment.Mar 16 2021, 1:26 PM

Change 672660 merged by Jbond:
[operations/puppet@production] puppet_compiler: stop installing openjdk

https://gerrit.wikimedia.org/r/672660

gerritbot added a comment.Mar 16 2021, 2:49 PM

Change 672658 merged by Muehlenhoff:
[operations/puppet@production] contint: use Java 11 on Jenkins agents

https://gerrit.wikimedia.org/r/672658

hashar added a comment.Mar 16 2021, 4:25 PM

I have some trouble to set the java version for Jenkins agents using hiera. The spin off task is T277526#6916812

gerritbot added a comment.Mar 18 2021, 2:45 PM

Change 673286 had a related patch set uploaded (by Hashar; owner: Hashar):
[operations/puppet@production] contint: remove erroneous hiera setting for labs

https://gerrit.wikimedia.org/r/673286

hashar added a comment.Mar 18 2021, 3:05 PM

Via Horizon ( https://horizon.wikimedia.org/project/puppet/ ) I have set:

profile::java::java_packages:
- variant: jre-headless
  version: "11"

On integration and puppet-diffs projects.

For the deployment-prep project, I have set it in https://horizon.wikimedia.org/project/prefixpuppet/ for the prefix deployment-deploy so that would only affect the deployment servers.

Stashbot added a comment.Mar 18 2021, 3:52 PM

Mentioned in SAL (#wikimedia-releng) [2021-03-18T15:52:44Z] <hashar> Disconnecting a bunch of Jenkins agents to upgrade them to Java 11 # T269354

hashar closed this task as Resolved.Mar 18 2021, 4:48 PM

All agents switched:

Node 'compiler1001.puppet-diffs.eqiad.wmflabs' is running 11.0.9.1+1-post-Debian-1deb10u2
Node 'compiler1002.puppet-diffs.eqiad.wmflabs' is running 11.0.9.1+1-post-Debian-1deb10u2
Node 'compiler1003.puppet-diffs.eqiad.wmflabs' is running 11.0.9.1+1-post-Debian-1deb10u2
Node 'contint1001' is running 11.0.9.1+1-post-Debian-1deb10u2
Node 'contint2001' is running 11.0.9.1+1-post-Debian-1deb10u2
Node 'deployment-deploy01' is running 11.0.6+10-post-Debian-1bpo91
Node 'integration-agent-docker-1001' is running 11.0.6+10-post-Debian-1bpo91
Node 'integration-agent-docker-1002' is running 11.0.6+10-post-Debian-1bpo91
Node 'integration-agent-docker-1003' is running 11.0.6+10-post-Debian-1bpo91
Node 'integration-agent-docker-1004' is running 11.0.6+10-post-Debian-1bpo91
Node 'integration-agent-docker-1005' is running 11.0.6+10-post-Debian-1bpo91
Node 'integration-agent-docker-1006' is running 11.0.6+10-post-Debian-1bpo91
Node 'integration-agent-docker-1007' is running 11.0.6+10-post-Debian-1bpo91
Node 'integration-agent-docker-1008' is running 11.0.6+10-post-Debian-1bpo91
Node 'integration-agent-docker-1009' is currently offline - skipping check
Node 'integration-agent-docker-1010' is running 11.0.6+10-post-Debian-1bpo91
Node 'integration-agent-docker-1011' is running 11.0.6+10-post-Debian-1bpo91
Node 'integration-agent-docker-1012' is running 11.0.6+10-post-Debian-1bpo91
Node 'integration-agent-docker-1013' is running 11.0.6+10-post-Debian-1bpo91
Node 'integration-agent-docker-1014' is running 11.0.6+10-post-Debian-1bpo91
Node 'integration-agent-docker-1016' is running 11.0.6+10-post-Debian-1bpo91
Node 'integration-agent-docker-1020' is running 11.0.6+10-post-Debian-1bpo91
Node 'integration-agent-pkgbuilder-1001' is running 11.0.9.1+1-post-Debian-1deb10u2
Node 'integration-agent-pkgbuilder-1002' is running 11.0.9.1+1-post-Debian-1deb10u2
Node 'integration-agent-puppet-docker-1002' is running 11.0.6+10-post-Debian-1bpo91
Node 'integration-agent-qemu-1001' is running 11.0.6+10-post-Debian-1bpo91
Node 'integration-castor03' is running 11.0.6+10-post-Debian-1bpo91

I have removed openjdk-8 packages from all instances. The packages were already removed from the production machines contint2001 / contint1001 / releases1002 / releases2002.

So it is done!

Stashbot added a comment.Mar 18 2021, 4:48 PM

Mentioned in SAL (#wikimedia-releng) [2021-03-18T16:48:58Z] <hashar> Purging openjdk-8 packages from Jenkins agents # T269354

gerritbot added a comment.Jul 16 2021, 2:30 PM

Change 673286 merged by Dzahn:

[operations/puppet@production] contint: remove erroneous hiera setting for labs

https://gerrit.wikimedia.org/r/673286

MoritzMuehlenhoff mentioned this in T313832: contint1002 service implementation tracking.Dec 5 2022, 12:34 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL