Page MenuHomePhabricator

Define generic job that runs unit tests within a Docker container
Closed, ResolvedPublic

Description

As a way to better evaluate how we might manage Dockerfiles and base images in a Docker CI world, we should experiment with the simple use case of running malu unit tests via Jenkins within a Docker container. The job could be as simple as spinning up a container using the repo's Dockerfile and executing a generic entrypoint (e.g. make test).

Event Timeline

dduvall created this task.Nov 11 2016, 1:01 AM
dduvall removed dduvall as the assignee of this task.Nov 11 2016, 1:07 AM

We could use harbormaster directly and bypass jenkins?

dduvall renamed this task from Define scap/tox job that runs unit tests within a Docker container to Define generic job that runs unit tests within a Docker container.Nov 14 2016, 10:52 PM
dduvall updated the task description. (Show Details)

We could use harbormaster directly and bypass jenkins?

I'd be interesting in experimenting with that, actually. For this first pass though, I'd like to simply use the CloudBees Docker Build Jenkins plugin.

dduvall added a comment.EditedNov 14 2016, 10:56 PM

@mmodell, I don't have access to the Phab credential necessary to create the Harbormaster build for this. Can you create one based off the existing build? (https://phabricator.wikimedia.org/harbormaster/plan/9/) It should invoke the differential-docker-test parameterized Jenkins build in the same exact way.

Also, I think I'm going to test this using the malu project instead so that I don't cause too much scap noise.

rGMALU is now tagged with meta-ci-docker-diffs which should trigger differential-docker-test via Plan 11

dduvall claimed this task.Nov 16 2016, 7:20 PM
dduvall triaged this task as Medium priority.

First successful runs:
https://integration.wikimedia.org/ci/job/differential-docker-test/27/console – run with complete rebuild including download of base image ~ 135 seconds of overhead
https://integration.wikimedia.org/ci/job/differential-docker-test/25/console – run with image build on cached base image ~ 83 seconds of overhead
https://integration.wikimedia.org/ci/job/differential-docker-test/26/console – run with fully cached image (includes npm dependencies) ~ 6 seconds of overhead

I was running into all sorts of permission related issues using the Jenkins plugin, since it relies on mounting the entire Jenkins home directory and temp directories and then executes docker run with a different --user than it specifies when the image is built. With that setup, it was nearly impossible to implement Docker commands before the main entry point that would both warm the NPM cache in a way that plays nice with Docker's union FS based caching, and in a way that wouldn't cause some directories/files to be owned by root before the main test execution.

I opted instead for a very minimal and completely generalized shell script—not using the plugin at all but does of course require docker—that does the following:

  1. Builds an image named after the Jenkins build tag based on the repo's Dockerfile.ci
    • In this experiment, the pre-build cache warming and environment setup is implemented directly in the repo's Dockerfile.ci (see D455: Testing Docker based CI) but some of this could just as easily be included in a base image that we maintain for Node.js applications, one that uses our production Node.js packages, etc. Or not; it's flexible.
    • The Dockerfile.ci in this case uses COPY to move all repo files into the container instead of relying on a shared volume. This approach introduces more overhead but avoided all of the permissions related pitfalls, and it keeps the execution completely isolated.
  2. Runs a new container based off that image, and simply delegates to whatever ENTRYPOINT has been declared in the Dockerfile.ci
  3. Uses docker diff to identify any files that have been newly created and copies them back to the host workspace so they can be archived as Jenkins artifacts
  4. Removes the container

This is super cool! <3

Dan gave me a one hour crash course in that POC and I must say *I am impressed.* That immediately address a bunch of caching related issues we have (huge got repos, package managers).

I am not entirely sold on the sequence of actions and how snapshots are invalidated. Will need a bit more experimentation to get something that is rock solid.

I will try to reproduce on my local machine, probably using MediaWiki has a play doll. Will see.

Kudos Dan and thanks Mukunda foenthe Harbor master plan !

dduvall closed this task as Resolved.Dec 15 2016, 8:45 PM