Page MenuHomePhabricator
Create Task
Maniphest T259646

Add Java 11 Image to Config Docker Images
Closed, ResolvedPublic
Actions

Description

Sonar cloud will no longer run with Java 8 in October 2020, details here: https://sonarcloud.io/documentation/appendices/end-of-support/

We need to update the Sonar to run in Java 11 but there is no Java 11 image currently in the Config images.

Notes from IRC:

  • The ci images comes from releng/java8 which comes from releng/ci-stretch
  • There is no Java 11 available for stretch, so we need to bump up to buster
  • We should rename the Java 8 CI image to Maven (suggested by @JAllemandou)
    • And rename the java8-sonar-scanner to sonar-scanner and it should inherit from releng/maven from above

To support each of Java 8 and Java 11 we can either:
)

  • Have a single Buster image that has Java 11 (from Debian) and Java8 (from component/jdk8). The JAVA_HOME env variable can be used to switch between Java versions.
    • One would need to specific the proper JAVA_HOME somehow.
    • Cause the images to be rather large (two jdk installed)
  • A common maven image that has sh entry point and then child images for each of java 11 and java 8
    • A user or the CI job then decide which java version to use based on the image name (example: maven-jdk8, maven-jdk11)

Details

SubjectRepoBranchLines +/-
sonar-scanner: Bump image version to use Java 11mediawiki/services/push-notificationsmaster+1 -1
jjb: update codehealth jobs to Java11integration/configmaster+3 -3
dockerfiles: rename sonar-scanner and use java11integration/configmaster+12 -5
dockerfiles: change base image of java8-* imagesintegration/configmaster+40 -10
dockerfiles: maven-java8 and maven-java11integration/configmaster+50 -3 K
dockerfiles: use a scratch image for mavenintegration/configmaster+3 K -0
dockerfiles: "move" java8 to maven-java8integration/configmaster+197 -31
dockerfiles: java 11 imageintegration/configmaster+13 -0
Customize query in gerrit

Related Objects

Event Timeline

Mstyles created this task.Aug 4 2020, 5:25 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 4 2020, 5:25 PM
hashar added projects: Continuous-Integration-Config, Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1)), Code-Health.Aug 4 2020, 6:25 PM
hashar updated the task description. (Show Details)Aug 4 2020, 6:29 PM

For Java 11 we have to base the releng/java8-sonar-scanner to be on Buster and we should also rename it to drop the java prefix from the image name.

While at it, I would to take the opportunity to add java 11 in the CI images. I am probably in favor of having two separate images (maven-jdk8 and maven-jdk11). That seems simpler than having to mess up with JAVA_HOME to pick a different java version than the default one.

hashar triaged this task as Medium priority.Aug 4 2020, 6:31 PM
kostajh subscribed.Aug 13 2020, 11:10 AM
hashar moved this task from INBOX to New Work on the Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1)) board.Aug 19 2020, 5:09 PM
Mstyles merged a task: T262821: SonarCloud analysis with Java 8 no longer supported after 2020-10-20.Sep 14 2020, 3:41 PM
Mstyles added a project: Discovery-Search.
Mstyles moved this task from needs triage to watching / waiting on the Discovery-Search board.
Mstyles added a subscriber: zeljkofilipin.
• Zbyszko subscribed.Sep 24 2020, 2:37 PM
Gehel added a comment.Sep 24 2020, 2:39 PM

Note that for Java projects, we want to continue to do the main build under Java 8, but the analysis with Java 11.

Jrbranaa added subscribers: thcipriani, greg, Jrbranaa.Sep 29 2020, 11:16 PM

@thcipriani and @greg this is the task that prompted the SonarCloud product ownership discussion and subsequent email to y'all.

zeljkofilipin added projects: Quality-and-Test-Engineering-Team, User-zeljkofilipin.Oct 6 2020, 11:23 AM
zeljkofilipin moved this task from Backlog 🪒 to Q4 👔 on the User-zeljkofilipin board.
hashar added a comment.Oct 6 2020, 4:42 PM

@zeljkofilipin I am on train duty this week. I can probably just do it, but I guess we can pair it next week? So at least you will be slightly familiar with how we build the Docker image which might prove helpful in the long run :]

zeljkofilipin added a comment.Oct 7 2020, 3:53 PM

@hashar Yes, please! Let me know when ever you're ready to work on this. I would be more than glad to see how it's done.

zeljkofilipin moved this task from Q4 👔 to Waiting 🏹 on the User-zeljkofilipin board.Oct 7 2020, 3:54 PM
hashar claimed this task.Oct 12 2020, 11:58 AM

I will refactor the images in order to share material between the java8 and java11 version and add a layer of images with just java 8 and java 11.

gerritbot added a comment.Oct 12 2020, 11:59 AM

Change 633523 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] dockerfiles: java 11 image

https://gerrit.wikimedia.org/r/633523

gerritbot added a project: Patch-For-Review.Oct 12 2020, 11:59 AM

Change 633524 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] dockerfiles: use a scratch image for maven

https://gerrit.wikimedia.org/r/633524

gerritbot added a comment.Oct 12 2020, 12:25 PM

Change 633523 merged by jenkins-bot:
[integration/config@master] dockerfiles: java 11 image

https://gerrit.wikimedia.org/r/633523

zeljkofilipin moved this task from Waiting 🏹 to Deep work 🌊 on the User-zeljkofilipin board.Oct 12 2020, 12:25 PM
gerritbot added a comment.Oct 12 2020, 12:49 PM

Change 633531 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] dockerfiles: "move" java8 to maven-java8

https://gerrit.wikimedia.org/r/633531

gerritbot added a comment.Oct 12 2020, 1:04 PM

Change 633532 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] dockerfiles: maven-java8 and maven-java11

https://gerrit.wikimedia.org/r/633532

gerritbot added a comment.Oct 12 2020, 1:34 PM

Change 633531 merged by jenkins-bot:
[integration/config@master] dockerfiles: "move" java8 to maven-java8

https://gerrit.wikimedia.org/r/633531

gerritbot added a comment.Oct 12 2020, 1:49 PM

Change 633524 merged by jenkins-bot:
[integration/config@master] dockerfiles: use a scratch image for maven

https://gerrit.wikimedia.org/r/633524

gerritbot added a comment.Oct 12 2020, 1:54 PM

Change 633532 merged by jenkins-bot:
[integration/config@master] dockerfiles: maven-java8 and maven-java11

https://gerrit.wikimedia.org/r/633532

Stashbot added a comment.Oct 12 2020, 1:58 PM

Mentioned in SAL (#wikimedia-releng) [2020-10-12T13:58:56Z] <hashar> Successfully tagged docker-registry.discovery.wmnet/releng/maven-java11:1.0.0 # T259646

Maintenance_bot removed a project: Patch-For-Review.Oct 12 2020, 2:10 PM
gerritbot added a comment.Oct 12 2020, 2:19 PM

Change 633549 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] dockerfiles: change base image of java8-* images

https://gerrit.wikimedia.org/r/633549

gerritbot added a project: Patch-For-Review.Oct 12 2020, 2:19 PM
gerritbot added a comment.Oct 12 2020, 2:21 PM

Change 633549 merged by jenkins-bot:
[integration/config@master] dockerfiles: change base image of java8-* images

https://gerrit.wikimedia.org/r/633549

gerritbot added a comment.Oct 12 2020, 2:59 PM

Change 633554 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] dockerfiles: rename sonar-scanner and use java11

https://gerrit.wikimedia.org/r/633554

gerritbot added a comment.Oct 12 2020, 3:03 PM

Change 633554 merged by jenkins-bot:
[integration/config@master] dockerfiles: rename sonar-scanner and use java11

https://gerrit.wikimedia.org/r/633554

gerritbot added a comment.Oct 12 2020, 3:21 PM

Change 633558 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] jjb: update codehealth jobs to Java11

https://gerrit.wikimedia.org/r/633558

Stashbot added a comment.Oct 12 2020, 3:22 PM

Mentioned in SAL (#wikimedia-releng) [2020-10-12T15:22:29Z] <hashar> Updated non voting codehealth jobs to Java11 for T259646: mwcore-codehealth-master-non-voting mwext-codehealth-master-non-voting

hashar added a comment.Oct 12 2020, 3:23 PM

I have updated the two non voting jobs to use the Java 11 based Docker images:

https://integration.wikimedia.org/ci/job/mwcore-codehealth-master-non-voting
https://integration.wikimedia.org/ci/job/mwext-codehealth-master-non-voting

If that looks ok we can do the rest I guess.

kostajh added a comment.Oct 12 2020, 7:33 PM
In T259646#6536885, @hashar wrote:

I have updated the two non voting jobs to use the Java 11 based Docker images:

https://integration.wikimedia.org/ci/job/mwcore-codehealth-master-non-voting
https://integration.wikimedia.org/ci/job/mwext-codehealth-master-non-voting

If that looks ok we can do the rest I guess.

👍 thanks!

hashar added a comment.Oct 12 2020, 7:52 PM

From a successful build of mwcore-codehealth-master-non-voting:

INFO: SonarQube Scanner 4.0.0.1744
INFO: Java 11.0.8 Debian (64-bit)
gerritbot added a comment.Oct 15 2020, 8:03 AM

Change 633558 merged by jenkins-bot:
[integration/config@master] jjb: update codehealth jobs to Java11

https://gerrit.wikimedia.org/r/633558

hashar closed this task as Resolved.Oct 15 2020, 8:07 AM

In order to support Java 11 in CI Docker image, I had to do a lot of various refactoring. When previously we had:

ci-stretch
 \-- java8
    \-- java8-sonar-scanner

We now have a hierarchy that looks like:

ci-stretch
 \-- java8
    \-- maven-java8
ci-buster
  \-- java11
    \-- maven-java11
      \-- sonar-scanner

And the actual change to fulfill that task ends up being straightforward: just change the parent image from maven-java8 to maven-java11 and drop the custom NodeJS that was required for Debian Stretch. https://gerrit.wikimedia.org/r/c/integration/config/+/633554/2/dockerfiles/sonar-scanner/Dockerfile.template

--- a/dockerfiles/java8-sonar-scanner/Dockerfile.template
+++ b/dockerfiles/sonar-scanner/Dockerfile.template
@@ -1,12 +1,11 @@
-FROM {{ "maven-java8" | image_tag }}
+FROM {{ "maven-java11" | image_tag }}
 
 USER root
 COPY KEYS /tmp/KEYS
 
 ENV SONAR_SCANNER_VERSION=4.0.0.1744
 
-RUN echo "deb http://apt.wikimedia.org/wikimedia stretch-wikimedia component/node10" > /etc/apt/sources.list.d/stretch-node10.list \
-    && {{ "nodejs gnupg wget unzip curl jq" | apt_install }} \
+RUN {{ "nodejs gnupg wget unzip curl jq" | apt_install }} \
     && git clone --depth 1 https://gerrit.wikimedia.org/r/p/integration/npm.git /srv/npm \
     && rm -rf /srv/npm/.git \
     && ln -s /srv/npm/bin/npm-cli.js /usr/bin/npm \
gerritbot added a comment.Oct 22 2020, 7:21 AM

Change 635744 had a related patch set uploaded (by Kosta Harlan; owner: Kosta Harlan):
[mediawiki/services/push-notifications@master] sonar-scanner: Bump image version to use Java 11

https://gerrit.wikimedia.org/r/635744

gerritbot added a comment.Oct 22 2020, 9:16 AM

Change 635744 merged by jenkins-bot:
[mediawiki/services/push-notifications@master] sonar-scanner: Bump image version to use Java 11

https://gerrit.wikimedia.org/r/635744

Don-vip mentioned this in T334512: Allow images from https://docker-registry.tools.wmflabs.org/ for CI under https://gitlab.wikimedia.org/toolforge-repos/.Apr 11 2023, 7:31 PM
Maintenance_bot removed a project: Patch-For-Review.Apr 11 2023, 8:11 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL