Related upstream: https://issues.jenkins-ci.org/browse/JENKINS-28175
Summary:
- Connection timeout for IRC
- Regular build timeout setting does not apply to post-build actions like artefacts and notifications.
- Cancelling the build doesn't work. It remains stuck.
- Disconnecting the slave and taking the slave offline has no effect.
- There seems to be no way to recover besides killing the whole Jenkins server and restarting.
The wmf-insecte Jenkins IRC bot no more show in channels.
We have logs at https://integration.wikimedia.org/ci/log/IRC%20IM%20plugin/
Jenkins full thread dump is at P584 and from there:
Two jobs are blocked:
"Executor #2 for integration-slave-trusty-1016 : executing browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-10-sauce #234" prio=5 BLOCKED "Executor #1 for integration-slave-trusty-1012 : executing browsertests-PdfHandler-test2.wikipedia.org-linux-firefox-sauce #494" prio=5 BLOCKED hudson.plugins.ircbot.v2.IRCConnectionProvider.getInstance(IRCConnectionProvider.java:14) hudson.plugins.ircbot.IrcPublisher.getIMConnection(IrcPublisher.java:102) hudson.plugins.im.IMPublisher.sendNotification(IMPublisher.java:374) hudson.plugins.im.IMPublisher.notifyChatsOnBuildEnd(IMPublisher.java:585) hudson.plugins.im.IMPublisher.notifyOnBuildEnd(IMPublisher.java:304) hudson.plugins.im.IMPublisher.perform(IMPublisher.java:291) ...
.A configuration submit change is blocked as well:
"Handling POST /ci/configSubmit from X.X.X.X : RequestHandlerThread[#1683]" daemon prio=5 WAITING sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303) java.util.concurrent.CountDownLatch.await(CountDownLatch.java:236) org.pircbotx.Channel.getMode(Channel.java:127) org.pircbotx.Channel.getModeArgument(Channel.java:182) org.pircbotx.Channel.getChannelKey(Channel.java:239) org.pircbotx.PircBotX.shutdown(PircBotX.java:2872) hudson.plugins.ircbot.v2.IRCConnection.close(IRCConnection.java:102) hudson.plugins.im.IMConnectionProvider.releaseConnection(IMConnectionProvider.java:92) hudson.plugins.ircbot.v2.IRCConnectionProvider.setDesc(IRCConnectionProvider.java:19) hudson.plugins.ircbot.IrcPublisher$DescriptorImpl.configure(IrcPublisher.java:336) jenkins.model.Jenkins.configureDescriptor(Jenkins.java:2915) jenkins.model.Jenkins.doConfigSubmit(Jenkins.java:2878) ...
Some other related threads:
"JenkinsIsBusyListener-thread" daemon prio=5 BLOCKED hudson.plugins.im.IMConnectionProvider.currentConnection(IMConnectionProvider.java:83) hudson.plugins.im.JenkinsIsBusyListener.setStatus(JenkinsIsBusyListener.java:118) hudson.plugins.im.JenkinsIsBusyListener.updateIMStatus(JenkinsIsBusyListener.java:109) hudson.plugins.im.JenkinsIsBusyListener.access$000(JenkinsIsBusyListener.java:20) hudson.plugins.im.JenkinsIsBusyListener$3.run(JenkinsIsBusyListener.java:98) java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) java.util.concurrent.FutureTask.run(FutureTask.java:262) java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178) java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745)
"IM-Reconnector-Thread" daemon prio=5 BLOCKED hudson.plugins.im.IMConnectionProvider$ConnectorRunnable.run(IMConnectionProvider.java:175) java.lang.Thread.run(Thread.java:745)