Page MenuHomePhabricator

Nodepool can't create slaves on Jenkins
Closed, DeclinedPublic

Description

When Nodepool attempts to create a Node in Jenkins over API, Jenkins emits a stacktrace.

Past upstream https://issues.jenkins-ci.org/browse/JENKINS-22762

Issue created: https://issues.jenkins-ci.org/browse/JENKINS-28993

https://integration.wikimedia.org/ci/computer/doCreateItem?json=%7B%22launcher%22%3A+%7B%22username%22%3A+%22nodepoolmanager%22%2C+%22stapler-class%22%3A+%22hudson.plugins.sshslaves.SSHLauncher%22%2C+%22host%22%3A+%2210.68.17.84%22%2C+%22privatekey%22%3A+%22%2Fvar%2Flib%2Fnodepool%2F.ssh%2Fnodepoolmanager_id_rsa%22%2C+%22port%22%3A+22%7D%2C+%22numExecutors%22%3A+1%2C+%22nodeProperties%22%3A+%7B%22stapler-class-bag%22%3A+%22true%22%7D%2C+%22name%22%3A+%22ci-jessie-wikimedia-33%22%2C+%22retentionStrategy%22%3A+%7B%22stapler-class%22%3A+%22hudson.slaves.RetentionStrategy%24Always%22%7D%2C+%22remoteFS%22%3A+%22%2Fhome%2Fjenkins%22%2C+%22type%22%3A+%22hudson.slaves.DumbSlave%24DescriptorImpl%22%2C+%22nodeDescription%22%3A+%22Dynamic+single+use+ci-jessie-wikimedia+node%22%2C+%22labelString%22%3A+%22ci-jessie-wikimedia%22%2C+%22mode%22%3A+%22EXCLUSIVE%22%7D&type=hudson.slaves.DumbSlave%24DescriptorImpl&name=ci-jessie-wikimedia-33

Which is == to:

https://integration.wikimedia.org/ci/computer/doCreateItem?
json=
{
    "labelString": "ci-jessie-wikimedia",
    "launcher": {
        "host": "10.68.17.84",
        "port": 22,
        "privatekey": "/var/lib/nodepool/.ssh/nodepoolmanager_id_rsa",
        "stapler-class": "hudson.plugins.sshslaves.SSHLauncher",
        "username": "nodepoolmanager"
    },  
    "mode": "EXCLUSIVE",
    "name": "ci-jessie-wikimedia-33",
    "nodeDescription": "Dynamic single use ci-jessie-wikimedia node",
    "nodeProperties": {
        "stapler-class-bag": "true"
    },  
    "numExecutors": 1,
    "remoteFS": "/home/jenkins",
    "retentionStrategy": {
        "stapler-class": "hudson.slaves.RetentionStrategy$Always"
    },  
    "type": "hudson.slaves.DumbSlave$DescriptorImpl"
}   
&type=hudson.slaves.DumbSlave$
DescriptorImpl&name=ci-jessie-wikimedia-33

Jenkins throws a stacktrace:

javax.servlet.ServletException: java.lang.RuntimeException: Failed to instantiate class hudson.slaves.DumbSlave from {"launcher":{"username":"nodepoolmanager","stapler-class":"hudson.plugins.sshslaves.SSHLauncher","host":"10.68.17.84","privatekey":"/var/lib/nodepool/.ssh/nodepoolmanager_id_rsa","port":22},"numExecutors":1,"nodeProperties":{"stapler-class-bag":"true"},"name":"ci-jessie-wikimedia-33","retentionStrategy":{"stapler-class":"hudson.slaves.RetentionStrategy$Always"},"remoteFS":"/home/jenkins","type":"hudson.slaves.DumbSlave$DescriptorImpl","nodeDescription":"Dynamic single use ci-jessie-wikimedia node","labelString":"ci-jessie-wikimedia","mode":"EXCLUSIVE"}
	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:796)
	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876)
	at org.kohsuke.stapler.MetaClass$4.doDispatch(MetaClass.java:211)
	at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746)
	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876)
	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:649)
	at org.kohsuke.stapler.Stapler.service(Stapler.java:238)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:686)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1494)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:96)
	at hudson.plugins.greenballs.GreenBallFilter.doFilter(GreenBallFilter.java:58)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:99)
	at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:198)
	at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:176)
	at net.bull.javamelody.PluginMonitoringFilter.doFilter(PluginMonitoringFilter.java:85)
	at org.jvnet.hudson.plugins.monitoring.HudsonMonitoringFilter.doFilter(HudsonMonitoringFilter.java:99)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:99)
	at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:88)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
	at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:48)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
	at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:93)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
	at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76)
	at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:168)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
	at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
	at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
	at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1474)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:499)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:533)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:428)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
	at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:68)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
	at org.eclipse.jetty.server.Server.handle(Server.java:366)
	at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
	at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:949)
	at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1011)
	at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:644)
	at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
	at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
	at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:668)
	at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
	at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: Failed to instantiate class hudson.slaves.DumbSlave from {"launcher":{"username":"nodepoolmanager","stapler-class":"hudson.plugins.sshslaves.SSHLauncher","host":"10.68.17.84","privatekey":"/var/lib/nodepool/.ssh/nodepoolmanager_id_rsa","port":22},"numExecutors":1,"nodeProperties":{"stapler-class-bag":"true"},"name":"ci-jessie-wikimedia-33","retentionStrategy":{"stapler-class":"hudson.slaves.RetentionStrategy$Always"},"remoteFS":"/home/jenkins","type":"hudson.slaves.DumbSlave$DescriptorImpl","nodeDescription":"Dynamic single use ci-jessie-wikimedia node","labelString":"ci-jessie-wikimedia","mode":"EXCLUSIVE"}
	at hudson.model.Descriptor.newInstance(Descriptor.java:577)
	at hudson.model.ComputerSet.doDoCreateItem(ComputerSet.java:293)
	at sun.reflect.GeneratedMethodAccessor7945.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:298)
	at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:161)
	at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:96)
	at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:121)
	at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746)
	... 72 more
Caused by: java.lang.NullPointerException
	at com.cloudbees.plugins.credentials.matchers.IdMatcher.<init>(IdMatcher.java:49)
	at com.cloudbees.plugins.credentials.CredentialsMatchers.withId(CredentialsMatchers.java:112)
	at hudson.plugins.sshslaves.SSHLauncher.lookupSystemCredentials(SSHLauncher.java:291)
	at hudson.plugins.sshslaves.SSHLauncher.<init>(SSHLauncher.java:268)
	at sun.reflect.GeneratedConstructorAccessor2257.newInstance(Unknown Source)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
	at org.kohsuke.stapler.RequestImpl.invokeConstructor(RequestImpl.java:447)
	at org.kohsuke.stapler.RequestImpl.instantiate(RequestImpl.java:699)
	at org.kohsuke.stapler.RequestImpl.access$200(RequestImpl.java:81)
	at org.kohsuke.stapler.RequestImpl$TypePair.convertJSON(RequestImpl.java:596)
	at org.kohsuke.stapler.RequestImpl.bindJSON(RequestImpl.java:400)
	at org.kohsuke.stapler.RequestImpl.instantiate(RequestImpl.java:693)
	at org.kohsuke.stapler.RequestImpl.access$200(RequestImpl.java:81)
	at org.kohsuke.stapler.RequestImpl$TypePair.convertJSON(RequestImpl.java:596)
	at org.kohsuke.stapler.RequestImpl.bindJSON(RequestImpl.java:400)
	at org.kohsuke.stapler.RequestImpl.bindJSON(RequestImpl.java:396)
	at hudson.model.Descriptor.newInstance(Descriptor.java:568)
	... 82 more

Event Timeline

hashar raised the priority of this task from to Needs Triage.
hashar updated the task description. (Show Details)
hashar added a subscriber: hashar.
hashar set Security to None.

Might want to pass some fake credential ID :(

hashar triaged this task as Lowest priority.Jun 19 2015, 7:42 PM
hashar updated the task description. (Show Details)
hashar added a project: Upstream.
hashar moved this task from Backlog to Reported Upstream on the Upstream board.
hashar moved this task from Backlog to Reported upstream on the Jenkins board.

I have reported the issue to upstream as https://issues.jenkins-ci.org/browse/JENKINS-28993

Anyway, the nodepool ssh key has to be in the Jenkins credential store so the master can actually SSH into the created images. So our bug is no more an issue.

hashar claimed this task.

It works just fine when given a credentials-id which is our use case. Leaving upstream bug open but there is no need for us to track this issue anymore.