Page MenuHomePhabricator

ERROR: Failed to notify endpoint 'HTTP:http://127.0.0.1:8001/jenkins_endpoint'
Closed, ResolvedPublic

Description

Consistently over the past 24 hours, whenever triggering a build (or rebuilding the last build), the following happens:

00:00:00.013 Notifying endpoint 'HTTP:http://127.0.0.1:8001/jenkins_endpoint'
00:00:00.022 ERROR: Failed to notify endpoint 'HTTP:http://127.0.0.1:8001/jenkins_endpoint'
00:00:00.022 java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
00:00:00.022 	at java.util.LinkedList.checkElementIndex(LinkedList.java:553)
00:00:00.022 	at java.util.LinkedList.set(LinkedList.java:488)
00:00:00.022 	at hudson.model.Run.getLog(Run.java:1981)
00:00:00.022 	at com.tikal.hudson.plugins.notification.Phase.getLog(Phase.java:141)
00:00:00.023 	at com.tikal.hudson.plugins.notification.Phase.buildJobState(Phase.java:79)
00:00:00.023 	at com.tikal.hudson.plugins.notification.Phase.handle(Phase.java:42)
00:00:00.023 	at com.tikal.hudson.plugins.notification.JobListener.onStarted(JobListener.java:31)
00:00:00.023 	at hudson.model.listeners.RunListener.fireStarted(RunListener.java:213)
00:00:00.023 	at hudson.model.Run.execute(Run.java:1755)
00:00:00.023 	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
00:00:00.024 	at hudson.model.ResourceController.execute(ResourceController.java:89)
00:00:00.024 	at hudson.model.Executor.run(Executor.java:240)
00:00:00.024 Failed to notify endpoint 'HTTP:http://127.0.0.1:8001/jenkins_endpoint' - java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
00:00:00.047 Building remotely on integration-slave1405 (phpflavor-hhvm contintLabsSlave UbuntuTrusty) in workspace /mnt/jenkins-workspace/workspace/mediawiki-core-qunit-karma

It does not cause the build to fail but is concerning nonetheless.

Presumably this is another one of those environment variables Zuul uses from Jenkins Job Builder that is not saved in the Build Parameters. However, contrary to those, this previously worked fine.

Probably caused by upstream Jenkins or Zuul which have both recently been upgraded.

Event Timeline

Krinkle assigned this task to hashar.
Krinkle raised the priority of this task from to Medium.
Krinkle updated the task description. (Show Details)
Krinkle added a subscriber: Krinkle.

Before Zuul migrated to Gearman, the jobs had to notify Zuul on start and completion. That is done using the Jenkins notification plugin and the configuration is done in JJB by using triggers: [ zuul ] which also adds the build parameters to the job.

Since Zuul migrated to Gearman, the jenkins_endpoint listener ( http://127.0.0.1:8001/jenkins_endpoint ) has been removed so those HTTPs requests are noops.

I have upgraded Jenkins yesterday around 11pm UTC, and probably have upgraded the notification plugin yesterday. There might be some incompatibility issue triggering the issue.

Anyway, since the notifications are noop, the best is to remove them entirely from JJB and refresh all our jobs.

I have proposed to upstream a patch in that sense: https://review.openstack.org/167254

Note: meanwhile the stracktrace is surely annoying but is not causing any harm.

Krinkle set Security to None.
hashar lowered the priority of this task from Medium to Low.Mar 26 2015, 11:37 AM
Krinkle renamed this task from Manually starting builds in Jenkins throws "java.lang.IndexOutOfBoundsException: Index: 0, Size: 0" to ERROR: Failed to notify endpoint 'HTTP:http://127.0.0.1:8001/jenkins_endpoint'.Apr 3 2015, 5:16 AM

Found an upstream bug report: https://issues.jenkins-ci.org/browse/JENKINS-27323 which states:

With the logLines config option, the default value is 0.
This causes an out of bounds issue. I had to manually change the value to -1 to address this.

I have downgraded the plugin from 1.9 to previous 1.7.

Before Zuul migrated to Gearman, the jobs had to notify Zuul on start and completion. That is done using the Jenkins notification plugin and the configuration is done in JJB by using triggers: [ zuul ] which also adds the build parameters to the job.

Since Zuul migrated to Gearman, the jenkins_endpoint listener ( http://127.0.0.1:8001/jenkins_endpoint ) has been removed so those HTTPs requests are noops.

I have upgraded Jenkins yesterday around 11pm UTC, and probably have upgraded the notification plugin yesterday. There might be some incompatibility issue triggering the issue.

Anyway, since the notifications are noop, the best is to remove them entirely from JJB and refresh all our jobs.

I have proposed to upstream a patch in that sense: https://review.openstack.org/167254

The JJB upstream patch has been merged. Our stalled copy of JJB in integration/jenkins-job-builder.git would need to be updated, it is lagging by a few months so that might have some side effects to be carefully checked.

Once updated, we will have to refresh all the jobs to get rid of the HTTP notification.

I did a first upgrade from 4bf020e..c492c0b

The next commit is 09bd39f which is https://review.openstack.org/#/c/148057/ . That changes the git plugin to use a new format. Not sure whether our current plugin version supports it nor whether we should update it.

There are a bunch of change that require us to bump our git plugin version. It is fairly updated and we should do it, then we can update JJB.

On Monday we bumped JJB. I am now bumping it to merge commit 4135e143 which is the patch I wrote https://review.openstack.org/#/c/148057/

Then refreshing all jobs.

hashar moved this task from In-progress to Done on the Continuous-Integration-Infrastructure board.

Jobs are still being refreshed but I confirmed the http notification is gone and jobs are properly triggered and reporting back to Gerrit. All good.