Page MenuHomePhabricator

Upgrade Jenkins Gearman plugin from a forked repo
Closed, ResolvedPublic

Description

The Jenkins Gearman plugin has originally be written for OpenStack (now OpenDev), they have phased out Jenkins from their architecture and the plugin has been unmaintained since. However, I have found out another company forked it and addressed a few issues with it, most notably Java 11. We should thus use their repository, rebuild our plugin and upgrade.

The rough topology of the git repositories:

https://github.com/jenkinsci/gearman-pluginDummy one just having a README
https://opendev.org/x/gearman-pluginOriginal canonical repository, the version WMF runs
https://github.com/gooddata/gearman-pluginMaintained fork

The Gearman plugin depends on gearman-java which does not work under Java 11 due to a breaking change https://www.oracle.com/java/technologies/javase/jdk-11-relnote.html#JDK-8200458

Event Timeline

Mentioned in SAL (#wikimedia-releng) [2021-01-11T08:59:51Z] <hashar> gerrit: created integration/jenkinsci/gearman-plugin.git to maintain the Jenkins Gearman plugin # T271683

I got it build with Java 8 simply after adding:

pom.xml
+    <repositories>
+        <repository>
+            <id>repo.jenkins-ci.org</id>
+            <url>https://repo.jenkins-ci.org/public/</url>
+        </repository>
+    </repositories>
+    <pluginRepositories>
+        <pluginRepository>
+            <id>repo.jenkins-ci.org</id>
+            <url>https://repo.jenkins-ci.org/public/</url>
+        </pluginRepository>
+    </pluginRepositories>

And then using: JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 mvn clean package


For Java 11 some Jenkins dependencies need to be updated:

pom.xml
--- a/pom.xml
+++ b/pom.xml
@@ -27,7 +27,7 @@
     <parent>
         <groupId>org.jenkins-ci.plugins</groupId>
         <artifactId>plugin</artifactId>
-        <version>3.55</version>
+        <version>3.57</version>
         <relativePath />
     </parent>
 
@@ -76,7 +76,7 @@
 
     <properties>
         <java.level>8</java.level>
-        <jenkins.version>2.107.2</jenkins.version>
+        <jenkins.version>2.164.1</jenkins.version>
         <!-- define all plugin versions -->
         <configuration-as-code.version>1.35</configuration-as-code.version>
         <gson.version>2.8.2</gson.version>

After that the tests fail:

org.mockito.exceptions.base.MockitoException: 
Mockito cannot mock this class: interface org.gearman.worker.GearmanWorker.
...
Java               : 11
...
Underlying exception : java.lang.UnsupportedOperationException: Cannot define class using reflection
...
Caused by: java.lang.IllegalStateException: Could not find sun.misc.Unsafe

Something something about Mockito / bitebuddy not supporting Java 11 I guess.

mvn dependency:tree
[INFO] +- org.mockito:mockito-core:jar:2.15.0:test
[INFO] |  +- net.bytebuddy:byte-buddy:jar:1.7.9:test
[INFO] |  \- net.bytebuddy:byte-buddy-agent:jar:1.7.9:test

The pom defines:

pom.xml
<properties>
          <mockito.version>2.15.0</mockito.version>
...
<dependencies>
          <dependency>
              <groupId>org.mockito</groupId>
              <artifactId>mockito-core</artifactId>
              <version>${mockito.version}</version>
              <scope>test</scope>
          </dependency>

The release notes at https://github.com/mockito/mockito/blob/release/2.x/doc/release-notes/official.md has a bunch of entries related to Java 11. I thus upgrade Mockito to the latest from 2.x branch: 2.28.2 and surely the build now passes with Java 11 (but fails on Java 8).

Change 655415 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/jenkinsci/gearman-plugin@master] Add back repo.jenkins-ci.org

https://gerrit.wikimedia.org/r/655415

Change 655416 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/jenkinsci/gearman-plugin@master] Update deps for Java 11

https://gerrit.wikimedia.org/r/655416

Change 655419 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] jjb: jobs for integration/jenkinsci/gearman-plugin

https://gerrit.wikimedia.org/r/655419

I have tried the Gearman plugin build for Java 11, it is unable to read the received data for some reason:

Session GearmanNIOJobServerConnection:localhost/127.0.0.1:4730 has read 0 bytes from its job server. Buffer has 32768
janv. 11, 2021 1:51:15 PM FINE org.gearman.common.GearmanNIOJobServerConnection read

However it does register the pipeline jobs and I have managed to get one build. So that is an improvement :]

After a full day of debugging:

Java 8Tests run: 76, Failures: 0, Errors: 0, Skipped: 0
Java 11Tests run: 76, Failures: 0, Errors: 0, Skipped: 0

The issue is in the gearman-java dependency which is on Launchpad (yeah Bazaar!) and hasn't been touched since ~ 2013.

--- src/main/java/org/gearman/common/GearmanJobServerSession.java	2011-02-01 21:26:50 +0000
+++ src/main/java/org/gearman/common/GearmanJobServerSession.java	2021-01-11 21:49:54 +0000
@@ -198,7 +198,7 @@
         while (canRead()) {
             p = connection.read();
             if (p == null) {
-                continue;
+                break;
             }
             handleSessionEvent(new GearmanSessionEvent(p, this));               //NOPMD
         }

Cause in Java 11, apparently the SelectorKey or connection is always readable (canRead() == true) and that leads to an infinite loop.

And might need a:

- orginalBuffer.flip();
+ ((Buffer)orginalBuffer).flip();

And of course update the dependencies:

pom.xml
--- pom.xml	2014-09-09 06:27:19 +0000
+++ pom.xml	2021-01-11 17:21:27 +0000
@@ -81,16 +81,12 @@
             <plugin>
                 <groupId>org.apache.maven.plugins</groupId>
                 <artifactId>maven-compiler-plugin</artifactId>
-                <configuration>
-                    <source>1.5</source>
-                    <target>1.5</target>
-                </configuration>
-                <version>2.4</version>
+                <version>3.8.1</version>
             </plugin>
             <plugin>
                 <groupId>org.apache.maven.plugins</groupId>
-                <artifactId>maven-surefire-plugin</artifactId>
-                <version>2.12</version>
+				<artifactId>maven-surefire-plugin</artifactId>
+				<version>3.0.0-M4</version>
             </plugin>
             <plugin>
                 <artifactId>maven-site-plugin</artifactId>
@@ -129,7 +125,7 @@
         <dependency>
             <groupId>junit</groupId>
             <artifactId>junit</artifactId>
-            <version>4.8</version>
+            <version>4.13.1</version>
             <scope>test</scope>
         </dependency>
         <dependency>
@@ -175,7 +171,7 @@
             <plugin>
               <groupId>org.apache.maven.plugins</groupId>
               <artifactId>maven-surefire-report-plugin</artifactId>
-              <version>2.12</version>
+              <version>3.0.0-M4</version>
             </plugin>
 	    <plugin>
 	      <groupId>org.codehaus.mojo</groupId>

As a nerd snipe, the test suite does not seem to pass when being run with the python-gear Gearman server. I have used the canonical C based server instead.

Stuff to do now:

  • Reach out to launchpad upstream authors
  • Migrate the repository to git (I have done that in the past for python-jenkins)
  • Patch it and ensure CI passes with jdk8 and jdk11
  • Find out how to release that
  • (stretch) Fix up python-gear
  • Point our fork of the Jenkins gearman-plugin to that forked release of gearman-java
  • non-profit!

Opensource is great, my java knowledge not that much :-\

Change 655663 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/gearman-java@master] Fix read infinite loop under Java 11

https://gerrit.wikimedia.org/r/655663

Change 656884 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] dockerfiles: images for integration/gearman-java

https://gerrit.wikimedia.org/r/656884

Change 656884 merged by jenkins-bot:
[integration/config@master] dockerfiles: images for integration/gearman-java

https://gerrit.wikimedia.org/r/656884

Change 656891 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Add integration/gearman-java

https://gerrit.wikimedia.org/r/656891

Change 656891 merged by jenkins-bot:
[integration/config@master] Add integration/gearman-java and Java 11 template

https://gerrit.wikimedia.org/r/656891

Change 657073 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] jjb: gearman-java jobs should timeout earlier

https://gerrit.wikimedia.org/r/657073

Change 657073 merged by jenkins-bot:
[integration/config@master] jjb: gearman-java jobs should timeout earlier

https://gerrit.wikimedia.org/r/657073

Change 657094 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] zuul: add java8 jobs for gearman-java

https://gerrit.wikimedia.org/r/657094

Change 657097 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] zuul: add java11 jobs for gearman-java

https://gerrit.wikimedia.org/r/657097

Change 657094 merged by jenkins-bot:
[integration/config@master] zuul: add java8 jobs for gearman-java

https://gerrit.wikimedia.org/r/657094

Change 657097 merged by jenkins-bot:
[integration/config@master] zuul: add java11 jobs for gearman-java

https://gerrit.wikimedia.org/r/657097

Change 657811 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/gearman-java@master] pom: move release to Wikimedia

https://gerrit.wikimedia.org/r/657811

Change 655663 merged by jenkins-bot:
[integration/gearman-java@master] Fix read infinite loop under Java 11

https://gerrit.wikimedia.org/r/655663

Change 657811 merged by jenkins-bot:
[integration/gearman-java@master] pom: move release to Wikimedia

https://gerrit.wikimedia.org/r/657811

Change 659860 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/jenkinsci/gearman-plugin@master] Switch gearman-java to Wikimedia fork

https://gerrit.wikimedia.org/r/659860

Change 655419 merged by jenkins-bot:
[integration/config@master] Add jobs for integration/jenkinsci/gearman-plugin

https://gerrit.wikimedia.org/r/655419

Change 655415 merged by Hashar:
[integration/jenkinsci/gearman-plugin@master] Add back repo.jenkins-ci.org

https://gerrit.wikimedia.org/r/655415

hashar triaged this task as Medium priority.Feb 5 2021, 1:37 PM

Change 664509 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] zuul: promote jobs for Jenkins Gearman plugin

https://gerrit.wikimedia.org/r/664509

Change 664509 merged by jenkins-bot:
[integration/config@master] zuul: promote jobs for Jenkins Gearman plugin

https://gerrit.wikimedia.org/r/664509

Change 655416 abandoned by Hashar:
[integration/jenkinsci/gearman-plugin@master] Update deps for Java 11

Reason:
Moved to https://github.com/jenkinsci/gearman-plugin/pull/7

https://gerrit.wikimedia.org/r/655416

Change 659860 abandoned by Hashar:
[integration/jenkinsci/gearman-plugin@master] Switch gearman-java to Wikimedia fork

Reason:
Moved to https://github.com/jenkinsci/gearman-plugin/pull/7

https://gerrit.wikimedia.org/r/659860

I have done a fairly large update to Wikitech which summarizes all the effort from this task: https://wikitech.wikimedia.org/wiki/Jenkins#Gearman_plugin

The plugin can be tested using:

A smoke test can be conducted on that instance. Create a freestyle job and a pipeline job.

Using python gear module (pip3 install --user gear):

import gear

client = gear.Client()
client.addServer('127.0.0.1')
client.waitForServer()

freestyle = gear.Job(b'build:freestyle', b'{}', '1')
client.submitJob(freestyle)

pipeline = gear.Job(b'build:pipeline', b'{}', '1')
client.submitJob(pipeline)

print('done')

That should have created a build for each of the job. Lot more should be tested such as asserting that parameters passed to the Gearman function make it to the job, but that is good enough to assert that the system works on Java 11.

Mentioned in SAL (#wikimedia-operations) [2021-02-22T07:14:36Z] <hashar> Restarting CI Jenkins for plugin upgrade # T271683

Mentioned in SAL (#wikimedia-operations) [2021-02-22T07:29:49Z] <hashar> Restarting CI Jenkins to downgrade plugin # T271683

Feb 22 07:28:06 contint2001 jenkins[4529]: SEVERE: [hudson.init.impl.InstallUncaughtExceptionHandler$DefaultUncaughtExceptionHandler uncaughtException] A thread (Gearman worker 172.17.0.1_manager/1017) died unexpectedly due to an uncaught exception, this may leave your Jenkins in a bad way and is usually indicative of a bug in the code.

And same for each of the workers. Fun times.

I got the trace from JavaMelody at https://integration.wikimedia.org/ci/monitoring

java.lang.NoSuchMethodError: java.nio.ByteBuffer.rewind()Ljava/nio/ByteBuffer;
      at org.gearman.common.GearmanNIOJobServerConnection.write(GearmanNIOJobServerConnection.java:154)
      at org.gearman.common.GearmanJobServerSession.driveSessionIO(GearmanJobServerSession.java:190)
      at hudson.plugins.gearman.MyGearmanWorkerImpl.registerFunctions(MyGearmanWorkerImpl.java:250)
      at hudson.plugins.gearman.MyGearmanWorkerImpl.work(MyGearmanWorkerImpl.java:345)
      at hudson.plugins.gearman.AbstractWorkerThread.run(AbstractWorkerThread.java:167)
      at java.lang.Thread.run(Thread.java:748)

We do use javac -source 8 -target 8:

pom.xml
<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-compiler-plugin</artifactId>
    <configuration>
        <source>1.8</source>
        <target>1.8</target>
    </configuration>
    <version>3.8.1</version>
</plugin>

But when compiling with Java 11, its java.* / javax.* are injected when compiling and the resulting bytecode points to non existent methods when run under Java 8.

There is a nice explanation at https://www.morling.dev/blog/bytebuffer-and-the-dreaded-nosuchmethoderror/ and further details at https://github.com/eclipse/jetty.project/issues/3244 . The trick is to instead use javac -release 8:

pom.xml
<properties>
  <maven.compiler.release>8</maven.compiler.release>
</properties>

Note that JDK 8 javac does not have a -release parameter so we might need a Maven profile to only activate it when compiling with JDK9 or later https://github.com/eclipse/jetty.project/blob/jetty-9.4.18.v20190429/pom.xml#L1843

I have released a fix with gearman-java 0.10.

I have properly tested the plugin with JDK 8, confirming it was not working with 0.9 but did with 0.10.

  • /usr/sbin/gearmand -l stderr -L 127.0.0.1 --verbose DEBUG
  • JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64/ mvn -Djenkins.version=2.263.3 hpi:run

And invoking a freestyle or a pipeline job through Gearman did work with the new gearman-java :]

Mentioned in SAL (#wikimedia-operations) [2021-02-23T07:13:18Z] <hashar> Restarting CI Jenkins for plugin upgrade # T271683

Seems to work properly this time! :-]