Page MenuHomePhabricator
Create Task
Maniphest T103120

Nodepool can't create slaves on Jenkins
Closed, DeclinedPublic
Actions

Description

When Nodepool attempts to create a Node in Jenkins over API, Jenkins emits a stacktrace.

Past upstream https://issues.jenkins-ci.org/browse/JENKINS-22762

Issue created: https://issues.jenkins-ci.org/browse/JENKINS-28993

https://integration.wikimedia.org/ci/computer/doCreateItem?json=%7B%22launcher%22%3A+%7B%22username%22%3A+%22nodepoolmanager%22%2C+%22stapler-class%22%3A+%22hudson.plugins.sshslaves.SSHLauncher%22%2C+%22host%22%3A+%2210.68.17.84%22%2C+%22privatekey%22%3A+%22%2Fvar%2Flib%2Fnodepool%2F.ssh%2Fnodepoolmanager_id_rsa%22%2C+%22port%22%3A+22%7D%2C+%22numExecutors%22%3A+1%2C+%22nodeProperties%22%3A+%7B%22stapler-class-bag%22%3A+%22true%22%7D%2C+%22name%22%3A+%22ci-jessie-wikimedia-33%22%2C+%22retentionStrategy%22%3A+%7B%22stapler-class%22%3A+%22hudson.slaves.RetentionStrategy%24Always%22%7D%2C+%22remoteFS%22%3A+%22%2Fhome%2Fjenkins%22%2C+%22type%22%3A+%22hudson.slaves.DumbSlave%24DescriptorImpl%22%2C+%22nodeDescription%22%3A+%22Dynamic+single+use+ci-jessie-wikimedia+node%22%2C+%22labelString%22%3A+%22ci-jessie-wikimedia%22%2C+%22mode%22%3A+%22EXCLUSIVE%22%7D&type=hudson.slaves.DumbSlave%24DescriptorImpl&name=ci-jessie-wikimedia-33

Which is == to:

https://integration.wikimedia.org/ci/computer/doCreateItem?
json=
{
    "labelString": "ci-jessie-wikimedia",
    "launcher": {
        "host": "10.68.17.84",
        "port": 22,
        "privatekey": "/var/lib/nodepool/.ssh/nodepoolmanager_id_rsa",
        "stapler-class": "hudson.plugins.sshslaves.SSHLauncher",
        "username": "nodepoolmanager"
    },  
    "mode": "EXCLUSIVE",
    "name": "ci-jessie-wikimedia-33",
    "nodeDescription": "Dynamic single use ci-jessie-wikimedia node",
    "nodeProperties": {
        "stapler-class-bag": "true"
    },  
    "numExecutors": 1,
    "remoteFS": "/home/jenkins",
    "retentionStrategy": {
        "stapler-class": "hudson.slaves.RetentionStrategy$Always"
    },  
    "type": "hudson.slaves.DumbSlave$DescriptorImpl"
}   
&type=hudson.slaves.DumbSlave$
DescriptorImpl&name=ci-jessie-wikimedia-33

Jenkins throws a stacktrace:

javax.servlet.ServletException: java.lang.RuntimeException: Failed to instantiate class hudson.slaves.DumbSlave from {"launcher":{"username":"nodepoolmanager","stapler-class":"hudson.plugins.sshslaves.SSHLauncher","host":"10.68.17.84","privatekey":"/var/lib/nodepool/.ssh/nodepoolmanager_id_rsa","port":22},"numExecutors":1,"nodeProperties":{"stapler-class-bag":"true"},"name":"ci-jessie-wikimedia-33","retentionStrategy":{"stapler-class":"hudson.slaves.RetentionStrategy$Always"},"remoteFS":"/home/jenkins","type":"hudson.slaves.DumbSlave$DescriptorImpl","nodeDescription":"Dynamic single use ci-jessie-wikimedia node","labelString":"ci-jessie-wikimedia","mode":"EXCLUSIVE"}
	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:796)
	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876)
	at org.kohsuke.stapler.MetaClass$4.doDispatch(MetaClass.java:211)
	at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746)
	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876)
	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:649)
	at org.kohsuke.stapler.Stapler.service(Stapler.java:238)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:686)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1494)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:96)
	at hudson.plugins.greenballs.GreenBallFilter.doFilter(GreenBallFilter.java:58)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:99)
	at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:198)
	at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:176)
	at net.bull.javamelody.PluginMonitoringFilter.doFilter(PluginMonitoringFilter.java:85)
	at org.jvnet.hudson.plugins.monitoring.HudsonMonitoringFilter.doFilter(HudsonMonitoringFilter.java:99)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:99)
	at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:88)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
	at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:48)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
	at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:93)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
	at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76)
	at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:168)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
	at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
	at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
	at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1474)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:499)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:533)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:428)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
	at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:68)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
	at org.eclipse.jetty.server.Server.handle(Server.java:366)
	at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
	at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:949)
	at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1011)
	at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:644)
	at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
	at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
	at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:668)
	at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
	at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: Failed to instantiate class hudson.slaves.DumbSlave from {"launcher":{"username":"nodepoolmanager","stapler-class":"hudson.plugins.sshslaves.SSHLauncher","host":"10.68.17.84","privatekey":"/var/lib/nodepool/.ssh/nodepoolmanager_id_rsa","port":22},"numExecutors":1,"nodeProperties":{"stapler-class-bag":"true"},"name":"ci-jessie-wikimedia-33","retentionStrategy":{"stapler-class":"hudson.slaves.RetentionStrategy$Always"},"remoteFS":"/home/jenkins","type":"hudson.slaves.DumbSlave$DescriptorImpl","nodeDescription":"Dynamic single use ci-jessie-wikimedia node","labelString":"ci-jessie-wikimedia","mode":"EXCLUSIVE"}
	at hudson.model.Descriptor.newInstance(Descriptor.java:577)
	at hudson.model.ComputerSet.doDoCreateItem(ComputerSet.java:293)
	at sun.reflect.GeneratedMethodAccessor7945.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:298)
	at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:161)
	at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:96)
	at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:121)
	at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746)
	... 72 more
Caused by: java.lang.NullPointerException
	at com.cloudbees.plugins.credentials.matchers.IdMatcher.<init>(IdMatcher.java:49)
	at com.cloudbees.plugins.credentials.CredentialsMatchers.withId(CredentialsMatchers.java:112)
	at hudson.plugins.sshslaves.SSHLauncher.lookupSystemCredentials(SSHLauncher.java:291)
	at hudson.plugins.sshslaves.SSHLauncher.<init>(SSHLauncher.java:268)
	at sun.reflect.GeneratedConstructorAccessor2257.newInstance(Unknown Source)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
	at org.kohsuke.stapler.RequestImpl.invokeConstructor(RequestImpl.java:447)
	at org.kohsuke.stapler.RequestImpl.instantiate(RequestImpl.java:699)
	at org.kohsuke.stapler.RequestImpl.access$200(RequestImpl.java:81)
	at org.kohsuke.stapler.RequestImpl$TypePair.convertJSON(RequestImpl.java:596)
	at org.kohsuke.stapler.RequestImpl.bindJSON(RequestImpl.java:400)
	at org.kohsuke.stapler.RequestImpl.instantiate(RequestImpl.java:693)
	at org.kohsuke.stapler.RequestImpl.access$200(RequestImpl.java:81)
	at org.kohsuke.stapler.RequestImpl$TypePair.convertJSON(RequestImpl.java:596)
	at org.kohsuke.stapler.RequestImpl.bindJSON(RequestImpl.java:400)
	at org.kohsuke.stapler.RequestImpl.bindJSON(RequestImpl.java:396)
	at hudson.model.Descriptor.newInstance(Descriptor.java:568)
	... 82 more

Event Timeline

hashar created this task.Jun 19 2015, 4:17 PM
hashar raised the priority of this task from to Needs Triage.
hashar updated the task description. (Show Details)
hashar added projects: Continuous-Integration-Scaling, Jenkins.
hashar subscribed.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 19 2015, 4:17 PM
hashar updated the task description. (Show Details)Jun 19 2015, 4:18 PM
hashar set Security to None.
hashar added a comment.Jun 19 2015, 4:20 PM

Might want to pass some fake credential ID :(

hashar added a comment.Jun 19 2015, 7:20 PM

Found the same trace on a couple sources:

https://github.com/arangamani/jenkins_api_client/issues/137 "Errors in creating a jenkins node" which ends up pointing to https://github.com/arangamani/jenkins_api_client/commit/b0b56221cf448ec9ddbb6bc0863754059eb4b1ef where the credential id is set.

On openstack paster: http://paste.openstack.org/show/90066/ from Aug 4, 2014 8:02:52 PM . Which lands to an IRC log:

http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2014-08-04.log
http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2014-08-04.log.html

A user having the same issue but no solution given :-D

hashar triaged this task as Lowest priority.Jun 19 2015, 7:42 PM
hashar updated the task description. (Show Details)
hashar added a project: Upstream.
hashar moved this task from Backlog to Reported Upstream on the Upstream board.
hashar moved this task from Backlog to Reported upstream on the Jenkins board.

I have reported the issue to upstream as https://issues.jenkins-ci.org/browse/JENKINS-28993

Anyway, the nodepool ssh key has to be in the Jenkins credential store so the master can actually SSH into the created images. So our bug is no more an issue.

hashar moved this task from Backlog to Done on the Continuous-Integration-Scaling board.Jun 19 2015, 7:43 PM
hashar closed this task as Declined.Jun 22 2015, 1:22 PM
hashar claimed this task.

It works just fine when given a credentials-id which is our use case. Leaving upstream bug open but there is no need for us to track this issue anymore.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL