Page MenuHomePhabricator

Jenkins does not shutdown due to a deadlock in the IRC plugin
Closed, ResolvedPublic

Description

Jenkins is still blocking on /etc/init.d/jenkins stop. I took a jstack for it:

Attaching to process ID 4217, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 24.75-b04
Deadlock Detection:

No deadlocks found.

Thread 31383: (state = BLOCKED)
 - java.lang.Shutdown.exit(int) @bci=96, line=212 (Interpreted frame)
 - java.lang.Terminator$1.handle(sun.misc.Signal) @bci=8, line=52 (Interpreted frame)
 - sun.misc.Signal$1.run() @bci=8, line=212 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 30581: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=186 (Interpreted frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt() @bci=1, line=834 (Interpreted frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(int) @bci=72, line=994 (Interpreted frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(int) @bci=24, line=1303 (Interpreted frame)
 - java.util.concurrent.CountDownLatch.await() @bci=5, line=236 (Interpreted frame)
 - org.pircbotx.Channel.getMode() @bci=76, line=127 (Interpreted frame)
 - org.pircbotx.Channel.getModeArgument(char) @bci=1, line=182 (Interpreted frame)
 - org.pircbotx.Channel.getChannelKey() @bci=3, line=239 (Interpreted frame)
 - org.pircbotx.PircBotX.shutdown(boolean) @bci=112, line=2872 (Interpreted frame)
 - org.pircbotx.PircBotX$4.run() @bci=38, line=2817 (Interpreted frame)


Thread 30574: (state = BLOCKED)
 - java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be imprecise)
 - java.lang.Thread.join(long) @bci=38, line=1281 (Interpreted frame)
 - java.lang.Thread.join() @bci=2, line=1355 (Interpreted frame)
 - java.lang.ApplicationShutdownHooks.runHooks() @bci=87, line=106 (Interpreted frame)
 - java.lang.ApplicationShutdownHooks$1.run() @bci=0, line=46 (Interpreted frame)
 - java.lang.Shutdown.runHooks() @bci=39, line=123 (Interpreted frame)
 - java.lang.Shutdown.sequence() @bci=26, line=167 (Interpreted frame)
 - java.lang.Shutdown.exit(int) @bci=96, line=212 (Interpreted frame)
 - java.lang.Terminator$1.handle(sun.misc.Signal) @bci=8, line=52 (Interpreted frame)
 - sun.misc.Signal$1.run() @bci=8, line=212 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 19094: (state = IN_NATIVE)
 - sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0 (Compiled frame; information may be imprecise)
 - sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=269 (Compiled frame)
 - sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=79 (Compiled frame)
 - sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=87 (Compiled frame)
 - sun.nio.ch.SelectorImpl.select(long) @bci=30, line=98 (Compiled frame)
 - sun.nio.ch.SelectorImpl.select() @bci=2, line=102 (Compiled frame)
 - hudson.plugins.gearman.MyGearmanWorkerImpl.work() @bci=606, line=340 (Compiled frame)
 - hudson.plugins.gearman.AbstractWorkerThread.run() @bci=103, line=166 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 5377: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=186 (Interpreted frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=2043 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=442 (Compiled frame)
 - org.jenkinsci.plugins.ZMQEventPublisher.ZMQRunnable.run() @bci=4, line=95 (Compiled frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 5373: (state = IN_NATIVE)
 - sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0 (Interpreted frame)
 - sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=269 (Interpreted frame)
 - sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=79 (Interpreted frame)
 - sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=87 (Interpreted frame)
 - sun.nio.ch.SelectorImpl.select(long) @bci=30, line=98 (Interpreted frame)
 - zmq.Poller.run() @bci=153, line=202 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 5372: (state = IN_NATIVE)
 - sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0 (Interpreted frame)
 - sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=269 (Interpreted frame)
 - sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=79 (Interpreted frame)
 - sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=87 (Interpreted frame)
 - sun.nio.ch.SelectorImpl.select(long) @bci=30, line=98 (Interpreted frame)
 - zmq.Poller.run() @bci=153, line=202 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 4362: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=186 (Compiled frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=2043 (Compiled frame)
 - java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() @bci=24, line=1079 (Interpreted frame)
 - java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() @bci=1, line=807 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=156, line=1068 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=26, line=1130 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=615 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 4361: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=186 (Interpreted frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt() @bci=1, line=834 (Interpreted frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(int) @bci=72, line=994 (Interpreted frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(int) @bci=24, line=1303 (Interpreted frame)
 - java.util.concurrent.Semaphore.acquire() @bci=5, line=317 (Interpreted frame)
 - hudson.plugins.im.IMConnectionProvider$ConnectorRunnable.run() @bci=4, line=160 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 4227: (state = BLOCKED)


Thread 4334: (state = BLOCKED)
 - java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be imprecise)
 - java.lang.ref.ReferenceQueue.remove(long) @bci=44, line=135 (Compiled frame)
 - java.lang.ref.ReferenceQueue.remove() @bci=2, line=151 (Compiled frame)
 - sun.java2d.Disposer.run() @bci=3, line=145 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 4239: (state = BLOCKED)


Thread 4238: (state = BLOCKED)
 - java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be imprecise)
 - java.lang.ref.ReferenceQueue.remove(long) @bci=44, line=135 (Compiled frame)
 - java.lang.ref.ReferenceQueue.remove() @bci=2, line=151 (Compiled frame)
 - java.lang.ref.Finalizer$FinalizerThread.run() @bci=36, line=209 (Compiled frame)


Thread 4237: (state = BLOCKED)
 - java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be imprecise)
 - java.lang.Object.wait() @bci=2, line=503 (Compiled frame)
 - java.lang.ref.Reference$ReferenceHandler.run() @bci=46, line=133 (Compiled frame)

Seems like the IRC notification plugin deadlocks the shutdown waiting for org.pircbotx.Channel.getMode. Might be similar to T96183: Jenkins: Builds (for beta cluster and browser tests) are stuck forever if IRC notification failed.

Event Timeline

hashar raised the priority of this task from to Needs Triage.
hashar updated the task description. (Show Details)
hashar added a subscriber: hashar.

Upstream issue in Jenkins plugin. We need to report a bug.

I really think it is related to T96183 and added a comment upstream on JENKINS-28175.

hashar moved this task from Backlog to Reported upstream on the Jenkins board.
hashar claimed this task.

Might have been solved via IRC plugin 2.27.