Blubber was designed to be an uncoupled piece in our continuous delivery pipeline, focusing only on building image variants from repo-provided config. Let's dogfood it as a simple and generic CI test runner.
The generic job can also make use of our new pipelinelib Groovy library.
Change 449527 had a related patch set uploaded (by Dduvall; owner: Dduvall):
[integration/config@master] Provide a generic job for testing via Blubber
I've create the new job in Jenkins but it appears that the library loading fails due to permissions: https://integration.wikimedia.org/ci/job/blubber-test/1/console
Started by user dduvall Running in Durability level: MAX_SURVIVABILITY Loading library wikimedia-integration-pipelinelib@master Attempting to resolve master from remote references... > git --version # timeout=10 > git ls-remote -h https://gerrit.wikimedia.org/r/integration/pipelinelib # timeout=10 Found match: refs/heads/master revision de98e5e79abd435eaabe09b2538156216f5be212 Using checkout strategy: Specific revision java.nio.file.AccessDeniedException: /srv/ssd/jenkins at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84) at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:384) at java.nio.file.Files.createDirectory(Files.java:674) at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781) at java.nio.file.Files.createDirectories(Files.java:767) at hudson.FilePath.mkdirs(FilePath.java:3098) at hudson.FilePath.access$900(FilePath.java:209) at hudson.FilePath$Mkdirs.invoke(FilePath.java:1216) at hudson.FilePath$Mkdirs.invoke(FilePath.java:1212) at hudson.FilePath.act(FilePath.java:1042) at hudson.FilePath.act(FilePath.java:1025) at hudson.FilePath.mkdirs(FilePath.java:1208) at hudson.plugins.git.GitSCM.createClient(GitSCM.java:810) at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1180) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:113) at org.jenkinsci.plugins.workflow.libs.SCMSourceRetriever.doRetrieve(SCMSourceRetriever.java:112) at org.jenkinsci.plugins.workflow.libs.SCMSourceRetriever.retrieve(SCMSourceRetriever.java:84) at org.jenkinsci.plugins.workflow.libs.LibraryAdder.retrieve(LibraryAdder.java:153) at org.jenkinsci.plugins.workflow.libs.LibraryAdder.add(LibraryAdder.java:134) at org.jenkinsci.plugins.workflow.libs.LibraryDecorator$1.call(LibraryDecorator.java:125) at org.codehaus.groovy.control.CompilationUnit.applyToPrimaryClassNodes(CompilationUnit.java:1065) at org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:603) at org.codehaus.groovy.control.CompilationUnit.processPhaseOperations(CompilationUnit.java:581) at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:558) at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:298) at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:268) at groovy.lang.GroovyShell.parseClass(GroovyShell.java:688) at groovy.lang.GroovyShell.parse(GroovyShell.java:700) at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.doParse(CpsGroovyShell.java:133) at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.reparse(CpsGroovyShell.java:127) at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.parseScript(CpsFlowExecution.java:557) at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.start(CpsFlowExecution.java:518) at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:290) at hudson.model.ResourceController.execute(ResourceController.java:97) at hudson.model.Executor.run(Executor.java:429) org.codehaus.groovy.control.MultipleCompilationErrorsException: startup failed: WorkflowScript: Loading libraries failed 1 error at org.codehaus.groovy.control.ErrorCollector.failIfErrors(ErrorCollector.java:310) at org.codehaus.groovy.control.CompilationUnit.applyToPrimaryClassNodes(CompilationUnit.java:1085) at org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:603) at org.codehaus.groovy.control.CompilationUnit.processPhaseOperations(CompilationUnit.java:581) at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:558) at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:298) at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:268) at groovy.lang.GroovyShell.parseClass(GroovyShell.java:688) at groovy.lang.GroovyShell.parse(GroovyShell.java:700) at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.doParse(CpsGroovyShell.java:133) at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.reparse(CpsGroovyShell.java:127) at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.parseScript(CpsFlowExecution.java:557) at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.start(CpsFlowExecution.java:518) at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:290) at hudson.model.ResourceController.execute(ResourceController.java:97) at hudson.model.Executor.run(Executor.java:429) Finished: FAILURE
/srv/ssd/jenkins is found in the Jenkins configuration:
<hudson> <workspaceDir>/srv/ssd/jenkins/workspace/${ITEM_FULLNAME}</workspaceDir> <buildsDir>/srv/jenkins/builds/${ITEM_FULL_NAME}</buildsDir> ... </hudson>
Since Jenkins 2.121.2 T199448 , those parameters are passed to the command line via system properties:
java \
-Djenkins.model.Jenkins.buildsDir=/srv/jenkins/builds/${ITEM_FULL_NAME} \
-jar /usr/share/jenkins/jenkins.war \We haven't set workspaceDir which is solely for running jobs on the master instances as user jenkins. That is disabled to prevent jobs from running as the jenkins user.
The blubber-test has to be attached to a slave. You can probably use one of the permanent Jessie slaves with the label selector: DebianJessie && contintLabsSlave. Or in JJB:
- job: name: blubber-test node: "DebianJessie && contintLabsSlave"
I have tried:
--- a/jjb/service-pipeline.yaml +++ b/jjb/service-pipeline.yaml @@ -63,6 +63,7 @@ - job: name: 'blubber-test' defaults: global + node: ServicePipelineProduction project-type: pipeline parameters: *service-pipeline-parameters dsl: !include-raw:
But the pipeline jobs do not use the node selector :/ I tried to move the Library import inside the node { } block but that fails differently:
Running in Durability level: MAX_SURVIVABILITY
org.codehaus.groovy.control.MultipleCompilationErrorsException: startup failed:
WorkflowScript: 2: Unknown type: IMPORT at line: 2 column: 3. File: WorkflowScript @ line 2, column 3.
@Library('wikimedia-integration-pipelinelib') import org.wikimedia.integration.*
^
1 errorSo I am not sure. I would really prefer we do not activate the master node on the CI Jenkins. It is too easy to have an uncontrolled job to roam on that node and run as the jenkins user :-\ However maybe the pipeline job does require execution to happen on the master, at least to parse the pipeline scripts / libraries.
Right! See T200953: Jenkins fails to checkout shared Groovy library integration/pipelinelib and the associated patch.
We haven't set workspaceDir which is solely for running jobs on the master instances as user jenkins. That is disabled to prevent jobs from running as the jenkins user.
Unfortunately, the workspace is also required for checking out shared library code, so we need it enabled for that to work.
The blubber-test has to be attached to a slave. You can probably use one of the permanent Jessie slaves with the label selector: DebianJessie && contintLabsSlave. Or in JJB:
- job: name: blubber-test node: "DebianJessie && contintLabsSlave"
Pipeline jobs must specify node labels in a different way. See the Groovy code in the associated patchset.
Change 449527 merged by Dduvall:
[integration/config@master] Provide a generic job for testing via Blubber
Change 451696 had a related patch set uploaded (by Dduvall; owner: Dduvall):
[integration/config@master] Test blubber and integration/pipelinelib projects
Change 451696 merged by jenkins-bot:
[integration/config@master] Test blubber and integration/pipelinelib projects
Both blubber and integration/pipelinelib are being tested by the new job upon new patches.