Mentioned in SAL (#wikimedia-releng) [2021-01-11T08:59:51Z] <hashar> gerrit: created integration/jenkinsci/gearman-plugin.git to maintain the Jenkins Gearman plugin # T271683

I got it build with Java 8 simply after adding:

+    <repositories>
+        <repository>
+            <id>repo.jenkins-ci.org</id>
+            <url>https://repo.jenkins-ci.org/public/</url>
+        </repository>
+    </repositories>
+    <pluginRepositories>
+        <pluginRepository>
+            <id>repo.jenkins-ci.org</id>
+            <url>https://repo.jenkins-ci.org/public/</url>
+        </pluginRepository>
+    </pluginRepositories>

And then using: JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 mvn clean package

For Java 11 some Jenkins dependencies need to be updated:

--- a/pom.xml
+++ b/pom.xml
@@ -27,7 +27,7 @@
     <parent>
         <groupId>org.jenkins-ci.plugins</groupId>
         <artifactId>plugin</artifactId>
-        <version>3.55</version>
+        <version>3.57</version>
         <relativePath />
     </parent>
 
@@ -76,7 +76,7 @@
 
     <properties>
         <java.level>8</java.level>
-        <jenkins.version>2.107.2</jenkins.version>
+        <jenkins.version>2.164.1</jenkins.version>
         <!-- define all plugin versions -->
         <configuration-as-code.version>1.35</configuration-as-code.version>
         <gson.version>2.8.2</gson.version>

After that the tests fail:

org.mockito.exceptions.base.MockitoException: 
Mockito cannot mock this class: interface org.gearman.worker.GearmanWorker.
...
Java               : 11
...
Underlying exception : java.lang.UnsupportedOperationException: Cannot define class using reflection
...
Caused by: java.lang.IllegalStateException: Could not find sun.misc.Unsafe

Something something about Mockito / bitebuddy not supporting Java 11 I guess.

[INFO] +- org.mockito:mockito-core:jar:2.15.0:test
[INFO] |  +- net.bytebuddy:byte-buddy:jar:1.7.9:test
[INFO] |  \- net.bytebuddy:byte-buddy-agent:jar:1.7.9:test

The pom defines:

<properties>
          <mockito.version>2.15.0</mockito.version>
...
<dependencies>
          <dependency>
              <groupId>org.mockito</groupId>
              <artifactId>mockito-core</artifactId>
              <version>${mockito.version}</version>
              <scope>test</scope>
          </dependency>

The release notes at https://github.com/mockito/mockito/blob/release/2.x/doc/release-notes/official.md has a bunch of entries related to Java 11. I thus upgrade Mockito to the latest from 2.x branch: 2.28.2 and surely the build now passes with Java 11 (but fails on Java 8).

Change 655415 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/jenkinsci/gearman-plugin@master] Add back repo.jenkins-ci.org

https://gerrit.wikimedia.org/r/655415

Change 655416 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/jenkinsci/gearman-plugin@master] Update deps for Java 11

https://gerrit.wikimedia.org/r/655416

Change 655419 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] jjb: jobs for integration/jenkinsci/gearman-plugin

https://gerrit.wikimedia.org/r/655419

I have tried the Gearman plugin build for Java 11, it is unable to read the received data for some reason:

Session GearmanNIOJobServerConnection:localhost/127.0.0.1:4730 has read 0 bytes from its job server. Buffer has 32768
janv. 11, 2021 1:51:15 PM FINE org.gearman.common.GearmanNIOJobServerConnection read

However it does register the pipeline jobs and I have managed to get one build. So that is an improvement :]

After a full day of debugging:

The issue is in the gearman-java dependency which is on Launchpad (yeah Bazaar!) and hasn't been touched since ~ 2013.

--- src/main/java/org/gearman/common/GearmanJobServerSession.java	2011-02-01 21:26:50 +0000
+++ src/main/java/org/gearman/common/GearmanJobServerSession.java	2021-01-11 21:49:54 +0000
@@ -198,7 +198,7 @@
         while (canRead()) {
             p = connection.read();
             if (p == null) {
-                continue;
+                break;
             }
             handleSessionEvent(new GearmanSessionEvent(p, this));               //NOPMD
         }

Cause in Java 11, apparently the SelectorKey or connection is always readable (canRead() == true) and that leads to an infinite loop.

And might need a:

- orginalBuffer.flip();
+ ((Buffer)orginalBuffer).flip();

And of course update the dependencies:

--- pom.xml	2014-09-09 06:27:19 +0000
+++ pom.xml	2021-01-11 17:21:27 +0000
@@ -81,16 +81,12 @@
             <plugin>
                 <groupId>org.apache.maven.plugins</groupId>
                 <artifactId>maven-compiler-plugin</artifactId>
-                <configuration>
-                    <source>1.5</source>
-                    <target>1.5</target>
-                </configuration>
-                <version>2.4</version>
+                <version>3.8.1</version>
             </plugin>
             <plugin>
                 <groupId>org.apache.maven.plugins</groupId>
-                <artifactId>maven-surefire-plugin</artifactId>
-                <version>2.12</version>
+				<artifactId>maven-surefire-plugin</artifactId>
+				<version>3.0.0-M4</version>
             </plugin>
             <plugin>
                 <artifactId>maven-site-plugin</artifactId>
@@ -129,7 +125,7 @@
         <dependency>
             <groupId>junit</groupId>
             <artifactId>junit</artifactId>
-            <version>4.8</version>
+            <version>4.13.1</version>
             <scope>test</scope>
         </dependency>
         <dependency>
@@ -175,7 +171,7 @@
             <plugin>
               <groupId>org.apache.maven.plugins</groupId>
               <artifactId>maven-surefire-report-plugin</artifactId>
-              <version>2.12</version>
+              <version>3.0.0-M4</version>
             </plugin>
 	    <plugin>
 	      <groupId>org.codehaus.mojo</groupId>

As a nerd snipe, the test suite does not seem to pass when being run with the python-gear Gearman server. I have used the canonical C based server instead.

Stuff to do now:

• Reach out to launchpad upstream authors
• Migrate the repository to git (I have done that in the past for python-jenkins)
• Patch it and ensure CI passes with jdk8 and jdk11
• Find out how to release that
• (stretch) Fix up python-gear
• Point our fork of the Jenkins gearman-plugin to that forked release of gearman-java
• non-profit!

Opensource is great, my java knowledge not that much :-\

Change 655663 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/gearman-java@master] Fix read infinite loop under Java 11

https://gerrit.wikimedia.org/r/655663

Change 656884 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] dockerfiles: images for integration/gearman-java

https://gerrit.wikimedia.org/r/656884

Change 656884 merged by jenkins-bot:
[integration/config@master] dockerfiles: images for integration/gearman-java

https://gerrit.wikimedia.org/r/656884

Change 656891 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Add integration/gearman-java

https://gerrit.wikimedia.org/r/656891

Change 656891 merged by jenkins-bot:
[integration/config@master] Add integration/gearman-java and Java 11 template

https://gerrit.wikimedia.org/r/656891

Change 657073 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] jjb: gearman-java jobs should timeout earlier

https://gerrit.wikimedia.org/r/657073

Change 657073 merged by jenkins-bot:
[integration/config@master] jjb: gearman-java jobs should timeout earlier

https://gerrit.wikimedia.org/r/657073

Change 657094 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] zuul: add java8 jobs for gearman-java

https://gerrit.wikimedia.org/r/657094

Change 657097 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] zuul: add java11 jobs for gearman-java

https://gerrit.wikimedia.org/r/657097

Change 657094 merged by jenkins-bot:
[integration/config@master] zuul: add java8 jobs for gearman-java

https://gerrit.wikimedia.org/r/657094

Change 657097 merged by jenkins-bot:
[integration/config@master] zuul: add java11 jobs for gearman-java

https://gerrit.wikimedia.org/r/657097

Change 657811 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/gearman-java@master] pom: move release to Wikimedia

https://gerrit.wikimedia.org/r/657811

Change 655663 merged by jenkins-bot:
[integration/gearman-java@master] Fix read infinite loop under Java 11

https://gerrit.wikimedia.org/r/655663

Change 657811 merged by jenkins-bot:
[integration/gearman-java@master] pom: move release to Wikimedia

https://gerrit.wikimedia.org/r/657811

Change 659860 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/jenkinsci/gearman-plugin@master] Switch gearman-java to Wikimedia fork

https://gerrit.wikimedia.org/r/659860

Change 655419 merged by jenkins-bot:
[integration/config@master] Add jobs for integration/jenkinsci/gearman-plugin

https://gerrit.wikimedia.org/r/655419

Change 655415 merged by Hashar:
[integration/jenkinsci/gearman-plugin@master] Add back repo.jenkins-ci.org

https://gerrit.wikimedia.org/r/655415

Change 664509 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] zuul: promote jobs for Jenkins Gearman plugin

https://gerrit.wikimedia.org/r/664509

Change 664509 merged by jenkins-bot:
[integration/config@master] zuul: promote jobs for Jenkins Gearman plugin

https://gerrit.wikimedia.org/r/664509

Change 655416 abandoned by Hashar:
[integration/jenkinsci/gearman-plugin@master] Update deps for Java 11

Reason:
Moved to https://github.com/jenkinsci/gearman-plugin/pull/7

https://gerrit.wikimedia.org/r/655416

Change 659860 abandoned by Hashar:
[integration/jenkinsci/gearman-plugin@master] Switch gearman-java to Wikimedia fork

Reason:
Moved to https://github.com/jenkinsci/gearman-plugin/pull/7

https://gerrit.wikimedia.org/r/659860

I have pushed a bunch of commits to https://github.com/jenkinsci/gearman-plugin/ and filed a request to get release rights https://github.com/jenkins-infra/repository-permissions-updater/pull/1830

I have done a fairly large update to Wikitech which summarizes all the effort from this task: https://wikitech.wikimedia.org/wiki/Jenkins#Gearman_plugin

The plugin can be tested using:

• https://github.com/jenkinsci/gearman-plugin/
• mvn -Djenkins.version=2.263.3 hpi:run
• xdg-open http://127.0.0.1:8080/jenkins/

A smoke test can be conducted on that instance. Create a freestyle job and a pipeline job.

Using python gear module (pip3 install --user gear):

import gear

client = gear.Client()
client.addServer('127.0.0.1')
client.waitForServer()

freestyle = gear.Job(b'build:freestyle', b'{}', '1')
client.submitJob(freestyle)

pipeline = gear.Job(b'build:pipeline', b'{}', '1')
client.submitJob(pipeline)

print('done')

That should have created a build for each of the job. Lot more should be tested such as asserting that parameters passed to the Gearman function make it to the job, but that is good enough to assert that the system works on Java 11.

I have tagged a new release https://github.com/jenkinsci/gearman-plugin/releases/tag/gearman-plugin-0.4.0 and it has been pushed to Jenkins artifactory https://repo.jenkins-ci.org/webapp/#/artifacts/browse/tree/General/releases/org/jenkins-ci/plugins/gearman-plugin/0.4.0/gearman-plugin-0.4.0.hpi

And it shows up in Jenkins center:

So we are one click away from breaki^Wupdating the plugin!

Mentioned in SAL (#wikimedia-operations) [2021-02-22T07:14:36Z] <hashar> Restarting CI Jenkins for plugin upgrade # T271683

Mentioned in SAL (#wikimedia-operations) [2021-02-22T07:29:49Z] <hashar> Restarting CI Jenkins to downgrade plugin # T271683

Feb 22 07:28:06 contint2001 jenkins[4529]: SEVERE: [hudson.init.impl.InstallUncaughtExceptionHandler$DefaultUncaughtExceptionHandler uncaughtException] A thread (Gearman worker 172.17.0.1_manager/1017) died unexpectedly due to an uncaught exception, this may leave your Jenkins in a bad way and is usually indicative of a bug in the code.

And same for each of the workers. Fun times.

I got the trace from JavaMelody at https://integration.wikimedia.org/ci/monitoring

java.lang.NoSuchMethodError: java.nio.ByteBuffer.rewind()Ljava/nio/ByteBuffer;
      at org.gearman.common.GearmanNIOJobServerConnection.write(GearmanNIOJobServerConnection.java:154)
      at org.gearman.common.GearmanJobServerSession.driveSessionIO(GearmanJobServerSession.java:190)
      at hudson.plugins.gearman.MyGearmanWorkerImpl.registerFunctions(MyGearmanWorkerImpl.java:250)
      at hudson.plugins.gearman.MyGearmanWorkerImpl.work(MyGearmanWorkerImpl.java:345)
      at hudson.plugins.gearman.AbstractWorkerThread.run(AbstractWorkerThread.java:167)
      at java.lang.Thread.run(Thread.java:748)

We do use javac -source 8 -target 8:

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-compiler-plugin</artifactId>
    <configuration>
        <source>1.8</source>
        <target>1.8</target>
    </configuration>
    <version>3.8.1</version>
</plugin>

But when compiling with Java 11, its java.* / javax.* are injected when compiling and the resulting bytecode points to non existent methods when run under Java 8.

There is a nice explanation at https://www.morling.dev/blog/bytebuffer-and-the-dreaded-nosuchmethoderror/ and further details at https://github.com/eclipse/jetty.project/issues/3244 . The trick is to instead use javac -release 8:

<properties>
  <maven.compiler.release>8</maven.compiler.release>
</properties>

Note that JDK 8 javac does not have a -release parameter so we might need a Maven profile to only activate it when compiling with JDK9 or later https://github.com/eclipse/jetty.project/blob/jetty-9.4.18.v20190429/pom.xml#L1843

I have released a fix with gearman-java 0.10.

I have properly tested the plugin with JDK 8, confirming it was not working with 0.9 but did with 0.10.

• /usr/sbin/gearmand -l stderr -L 127.0.0.1 --verbose DEBUG
• JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64/ mvn -Djenkins.version=2.263.3 hpi:run

And invoking a freestyle or a pipeline job through Gearman did work with the new gearman-java :]

gearman-java 0.10 addresses the ByteBuffer issue under JDK 8 https://archiva.wikimedia.org/#artifact-details-download-content~releases/org.wikimedia.gearman/gearman-java/0.10

And updated in the Jenkins plugin via https://github.com/jenkinsci/gearman-plugin/pull/10 which is in Gearman plugin 0.4.1

Mentioned in SAL (#wikimedia-operations) [2021-02-23T07:13:18Z] <hashar> Restarting CI Jenkins for plugin upgrade # T271683

Does the latest plugin depend on gearman-java?

• use gearmand(c binary version)
• Declared label as this in pipeline: agent {node label "foo"}

But the gearman server doesnot have build:Job:foo

@Tambura605 sorry I only seen your notification today (I receive way too many emails). Since I really had to write a summary of my actions here it is:

The Jenkins plugin was written for OpenStack which have abandoned Jenkins roughly in 2016 in favor of a new system. Some were familiar with the Gearman protocol since they previously worked at LiveJournal or similar and went to use the protocol they knew about.

The plugin being written in Java, they relied on a gearman-java which is no more maintained since 2014, hosted with bzr on launchpad and targeted Java 5. It worked well until Java 11 introduced a breaking change JDK-8200458.

I have converted the launchpad repo to a git repo hosted on Wikimedia Gerrit, fixed the code and released a new version (org.wikimedia.gearman:gearman-java:0.10).

For the Gearman plugin, OpenStack Infrastructure (now OpenDev) keeps an archive at https://opendev.org/x/gearman-plugin. I went to migrate that to the Jenkins CI GitHub organization to potentially benefit from shared maintenance. The canonical place repository is now https://github.com/jenkinsci/gearman-plugin/ and it now relies on the new gearman-java above which means the plugin supports Java 11. I have also merged in the patches that were still pending on OpenDev and merged in changes made to a fork at https://github.com/gooddata/gearman-plugin .

That forks bring support for Pipeline jobs https://github.com/gooddata/gearman-plugin . They are a bit specific since they are not assigned to any node, instead they execute on the primary Jenkins with a "one off executor" which is created on the fly even if the primary controller does not have any execution slot. The code creates a oneOfExecutor which runs on the primary, process the pipeline which then defines on which nodes/labels the build stages are executed.

On our system the primary controller does not have any executor on the primary controller, the Gearman Jenkins plugin thus does not register the jobs and they do not show up in the Gearman server. I think the relevant commit is https://github.com/gooddata/gearman-plugin/commit/3bf4b1a53f00acd2209e660cd3d9b83a8956fa54

If you create executors on the primary controller, that might cause the pipeline jobs to be registered. Probably has something like build:yourjob:built-in executor or something similar. Which only means one can potentially run a job on the primary controller executor and thus run as the Jenkins user. I haven't dig much into it though.

Answers to your questions in next comment.

@Tambura605 questions:

Does the latest plugin depend on gearman-java?

Yes, if you get the plugin from Jenkins plugin manager, you get the version created from GitHub jenkinsci/gearman-plugin.

The dependency is defined at https://github.com/jenkinsci/gearman-plugin/blob/c5c403c/pom.xml#L138-L153 and the version above at https://github.com/jenkinsci/gearman-plugin/blob/c5c403c/pom.xml#L92 :

<properties>
  <gearman.version>0.10</gearman.version>
</properties>
...
    <dependencies>
        <dependency>
            <groupId>org.wikimedia.gearman</groupId>
            <artifactId>gearman-java</artifactId>
            <version>${gearman.version}</version>
    </dependencies>

use gearmand(c binary version)

It is fine. We do not use the C germand. Our CI system (Zuul) spins up a gearman server implemented in Python https://pypi.org/project/gear/ . It is pretty much identical with one addition to the protocol to register a bulk of jobs in a single command (see https://github.com/gearman/gearmand/issues/6 and https://github.com/openstack-infra/zuul/commit/d43715988766fa95b88af5fc2c9d2c2aa723b4f9 ), but that addition is irrelevant to Jenkins which does not use that. So the gearmand C server should be fine.

Declared label as this in pipeline: agent {node label "foo"}
But the gearman server does not have build:Job:foo

Before updating the Gearman plugin to a version which supports Pipeline jobs, I have proposed to introduced normal freestyle jobs (which do get registered) which then triggers the Pipeline jobs https://gerrit.wikimedia.org/r/c/integration/config/+/391136/ That adds an extra layer but works.

The problem is the plugin only registers the Pipeline job to an executor on the primary controller. If there are no executor, the job does not get registered. See my comment at T271107#6882038 : the pipeline Jobs are always assigned to master and given it has no executor nothing get registered to Gearman.

You can try adding executors to the built-in node and see whether the Pipeline jobs get registered this way. They might even be buildable. I have no idea what the security implications are though.

I think on recent Jenkins the node is known as built-in so theoretically the registered function would be build:Job:built-in. The Pipeline job will be processed on the primary controler and then given you have agent {node label "foo"} Jenkins will run the pipeline on one of the matching agent(s).