Page MenuHomePhabricator

Generic CI job for running Blubber-built test entry points
Closed, ResolvedPublic

Description

Blubber was designed to be an uncoupled piece in our continuous delivery pipeline, focusing only on building image variants from repo-provided config. Let's dogfood it as a simple and generic CI test runner.

The generic job can also make use of our new pipelinelib Groovy library.

Details

Related Gerrit Patches:

Event Timeline

dduvall created this task.Jul 31 2018, 7:43 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 31 2018, 7:43 PM
dduvall claimed this task.Jul 31 2018, 7:43 PM
dduvall triaged this task as Medium priority.
dduvall moved this task from Backlog to CI on the Release Pipeline board.
dduvall moved this task from Backlog to In-progress on the Release-Engineering-Team (Kanban) board.

Change 449527 had a related patch set uploaded (by Dduvall; owner: Dduvall):
[integration/config@master] Provide a generic job for testing via Blubber

https://gerrit.wikimedia.org/r/449527

I've create the new job in Jenkins but it appears that the library loading fails due to permissions: https://integration.wikimedia.org/ci/job/blubber-test/1/console

Started by user dduvall
Running in Durability level: MAX_SURVIVABILITY
Loading library wikimedia-integration-pipelinelib@master
Attempting to resolve master from remote references...
 > git --version # timeout=10
 > git ls-remote -h https://gerrit.wikimedia.org/r/integration/pipelinelib # timeout=10
Found match: refs/heads/master revision de98e5e79abd435eaabe09b2538156216f5be212
Using checkout strategy: Specific revision
java.nio.file.AccessDeniedException: /srv/ssd/jenkins
	at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84)
	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
	at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:384)
	at java.nio.file.Files.createDirectory(Files.java:674)
	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781)
	at java.nio.file.Files.createDirectories(Files.java:767)
	at hudson.FilePath.mkdirs(FilePath.java:3098)
	at hudson.FilePath.access$900(FilePath.java:209)
	at hudson.FilePath$Mkdirs.invoke(FilePath.java:1216)
	at hudson.FilePath$Mkdirs.invoke(FilePath.java:1212)
	at hudson.FilePath.act(FilePath.java:1042)
	at hudson.FilePath.act(FilePath.java:1025)
	at hudson.FilePath.mkdirs(FilePath.java:1208)
	at hudson.plugins.git.GitSCM.createClient(GitSCM.java:810)
	at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1180)
	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:113)
	at org.jenkinsci.plugins.workflow.libs.SCMSourceRetriever.doRetrieve(SCMSourceRetriever.java:112)
	at org.jenkinsci.plugins.workflow.libs.SCMSourceRetriever.retrieve(SCMSourceRetriever.java:84)
	at org.jenkinsci.plugins.workflow.libs.LibraryAdder.retrieve(LibraryAdder.java:153)
	at org.jenkinsci.plugins.workflow.libs.LibraryAdder.add(LibraryAdder.java:134)
	at org.jenkinsci.plugins.workflow.libs.LibraryDecorator$1.call(LibraryDecorator.java:125)
	at org.codehaus.groovy.control.CompilationUnit.applyToPrimaryClassNodes(CompilationUnit.java:1065)
	at org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:603)
	at org.codehaus.groovy.control.CompilationUnit.processPhaseOperations(CompilationUnit.java:581)
	at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:558)
	at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:298)
	at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:268)
	at groovy.lang.GroovyShell.parseClass(GroovyShell.java:688)
	at groovy.lang.GroovyShell.parse(GroovyShell.java:700)
	at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.doParse(CpsGroovyShell.java:133)
	at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.reparse(CpsGroovyShell.java:127)
	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.parseScript(CpsFlowExecution.java:557)
	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.start(CpsFlowExecution.java:518)
	at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:290)
	at hudson.model.ResourceController.execute(ResourceController.java:97)
	at hudson.model.Executor.run(Executor.java:429)
org.codehaus.groovy.control.MultipleCompilationErrorsException: startup failed:
WorkflowScript: Loading libraries failed

1 error

	at org.codehaus.groovy.control.ErrorCollector.failIfErrors(ErrorCollector.java:310)
	at org.codehaus.groovy.control.CompilationUnit.applyToPrimaryClassNodes(CompilationUnit.java:1085)
	at org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:603)
	at org.codehaus.groovy.control.CompilationUnit.processPhaseOperations(CompilationUnit.java:581)
	at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:558)
	at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:298)
	at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:268)
	at groovy.lang.GroovyShell.parseClass(GroovyShell.java:688)
	at groovy.lang.GroovyShell.parse(GroovyShell.java:700)
	at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.doParse(CpsGroovyShell.java:133)
	at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.reparse(CpsGroovyShell.java:127)
	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.parseScript(CpsFlowExecution.java:557)
	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.start(CpsFlowExecution.java:518)
	at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:290)
	at hudson.model.ResourceController.execute(ResourceController.java:97)
	at hudson.model.Executor.run(Executor.java:429)
Finished: FAILURE

/srv/ssd/jenkins is found in the Jenkins configuration:

/var/lib/jenkins/config.xml
<hudson>
  <workspaceDir>/srv/ssd/jenkins/workspace/${ITEM_FULLNAME}</workspaceDir>
  <buildsDir>/srv/jenkins/builds/${ITEM_FULL_NAME}</buildsDir>
  ...
</hudson>

Since Jenkins 2.121.2 T199448 , those parameters are passed to the command line via system properties:

java \
  -Djenkins.model.Jenkins.buildsDir=/srv/jenkins/builds/${ITEM_FULL_NAME} \
  -jar /usr/share/jenkins/jenkins.war \

We haven't set workspaceDir which is solely for running jobs on the master instances as user jenkins. That is disabled to prevent jobs from running as the jenkins user.

The blubber-test has to be attached to a slave. You can probably use one of the permanent Jessie slaves with the label selector: DebianJessie && contintLabsSlave. Or in JJB:

- job:
    name: blubber-test
    node: "DebianJessie && contintLabsSlave"

I have tried:

--- a/jjb/service-pipeline.yaml
+++ b/jjb/service-pipeline.yaml
@@ -63,6 +63,7 @@
 - job:
     name: 'blubber-test'
     defaults: global
+    node: ServicePipelineProduction
     project-type: pipeline
     parameters: *service-pipeline-parameters
     dsl: !include-raw:

But the pipeline jobs do not use the node selector :/ I tried to move the Library import inside the node { } block but that fails differently:

Running in Durability level: MAX_SURVIVABILITY
org.codehaus.groovy.control.MultipleCompilationErrorsException: startup failed:
WorkflowScript: 2: Unknown type: IMPORT at line: 2 column: 3. File: WorkflowScript @ line 2, column 3.
     @Library('wikimedia-integration-pipelinelib') import org.wikimedia.integration.*
     ^

1 error

So I am not sure. I would really prefer we do not activate the master node on the CI Jenkins. It is too easy to have an uncontrolled job to roam on that node and run as the jenkins user :-\ However maybe the pipeline job does require execution to happen on the master, at least to parse the pipeline scripts / libraries.

/srv/ssd/jenkins is found in the Jenkins configuration:

/var/lib/jenkins/config.xml
<hudson>
  <workspaceDir>/srv/ssd/jenkins/workspace/${ITEM_FULLNAME}</workspaceDir>
  <buildsDir>/srv/jenkins/builds/${ITEM_FULL_NAME}</buildsDir>
  ...
</hudson>

Since Jenkins 2.121.2 T199448 , those parameters are passed to the command line via system properties:

java \
  -Djenkins.model.Jenkins.buildsDir=/srv/jenkins/builds/${ITEM_FULL_NAME} \
  -jar /usr/share/jenkins/jenkins.war \

Right! See T200953: Jenkins fails to checkout shared Groovy library integration/pipelinelib and the associated patch.

We haven't set workspaceDir which is solely for running jobs on the master instances as user jenkins. That is disabled to prevent jobs from running as the jenkins user.

Unfortunately, the workspace is also required for checking out shared library code, so we need it enabled for that to work.

The blubber-test has to be attached to a slave. You can probably use one of the permanent Jessie slaves with the label selector: DebianJessie && contintLabsSlave. Or in JJB:

- job:
    name: blubber-test
    node: "DebianJessie && contintLabsSlave"

Pipeline jobs must specify node labels in a different way. See the Groovy code in the associated patchset.

Change 449527 merged by Dduvall:
[integration/config@master] Provide a generic job for testing via Blubber

https://gerrit.wikimedia.org/r/449527

Change 451696 had a related patch set uploaded (by Dduvall; owner: Dduvall):
[integration/config@master] Test blubber and integration/pipelinelib projects

https://gerrit.wikimedia.org/r/451696

Change 451696 merged by jenkins-bot:
[integration/config@master] Test blubber and integration/pipelinelib projects

https://gerrit.wikimedia.org/r/451696

dduvall closed this task as Resolved.EditedAug 10 2018, 6:17 PM

Both blubber and integration/pipelinelib are being tested by the new job upon new patches.

https://integration.wikimedia.org/ci/blue/organizations/jenkins/blubber-test/detail/blubber-test/14/pipeline/17