Page MenuHomePhabricator

Define new Jenkins pipeline for container build phase
Closed, ResolvedPublic


We've sketched out a rough outline for the build phase of the container release pipeline. We should implement this as a Jenkins pipeline (as in the Jenkins Pipeline plugin) and see if we can't get something suitable for building Mathoid images.

  • blubber build test image
  • docker run test image entrypoint
  • decision/feedback fork:
    • test entrypoint passes
      • blubber build production image
      • push image to WMF docker registry
      • provide feedback
    • test entrypoint fails
      • abort pipeline
      • provide feedback

Event Timeline

Change 380551 had a related patch set uploaded (by Dduvall; owner: Dduvall):
[integration/config@master] WIP Service pipeline DSL

dduvall triaged this task as Medium priority.
dduvall moved this task from Backlog to CI on the Release Pipeline board.

An experimental job has been created from the current JJB patchset and is working up until just before the registry push.

Successfully built 9d25c234a7aa
Successfully tagged mediawiki-services-mathoid:build-20
[Pipeline] dockerFingerprintFrom
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline Cannot retrieve .Id from 'docker AS prep'
	at org.jenkinsci.plugins.docker.workflow.client.DockerClient.inspectRequiredField(
	at org.jenkinsci.plugins.docker.workflow.FromFingerprintStep$
	at org.jenkinsci.plugins.docker.workflow.FromFingerprintStep$
	at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1$
	at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$
	at java.util.concurrent.Executors$
	at java.util.concurrent.ThreadPoolExecutor.runWorker(
	at java.util.concurrent.ThreadPoolExecutor$
Finished: FAILURE

This failure may be due to an unresolved bug.

The more specifically relevant bug may be this one which indicates it's an issue with the FROM aliases used for multistage. The recommended workaround is to simply use sh to execute docker build for now. :) Easy enough.

Patchset 4 successfully builds the production image but fails with a 403 when trying to push to the registry. It's unclear whether our uploader credential is being used.

On integration-slave-docker-1705 I changed the docker.service to use debug log level (in ExecStart, pass -D) and journalctl -u docker --follow` to watch.

Then docker push and in the system log:

msg="Calling GET /_ping"
msg="Calling POST /v1.30/images/"
msg="hostDir: /etc/docker/certs.d/"
msg="Trying to push to v2"
msg="Pushing repository:"

Then there is a 403.

Using docker login and the credential works all fine though:

$ docker login
Username (uploader): uploader
Login Succeeded

Who knows whether docker POST with the credentials or if we get access for it :(

Clarified with @Joe labs instances are not allowed to interact with the docker registry. Nginx rejects them and that is based on the puppet bits:

class profile::docker::registry {
    $image_builders = hiera(
    class { '::docker::registry::web':
        allow_push_from      => $image_builders,

And in hiera:

profile::docker::registry::image_builders: [''] # copper.eqiad.wmnet

Hence the 403 is working as intended.

So I guess that has to be build on a production slave, potentially with a dedicated Jenkins instance?

dduvall changed the task status from Open to Stalled.Sep 27 2017, 5:00 PM

Right on. Thanks for debugging that @hashar! I think we can continue to test the experimental job on the labs instance and just push to Docker Hub for now. Once we get the dependencies installed on contint1001 (blubber, docker >= 17.05) and the job is somewhat stable, we can move it there.

Another issue that has arisen in testing this is that the default credential store for Docker is very insecure. By default, it stores all usernames/passwords given to docker login in the clear (in ~/.docker/config.json) until a subsequent docker logout is called. That means there is a substantial window of time where any other Jenkins job running via the same remote agent could gain access to the registry credentials. I'll open a subtask to further discuss and resolve this issue.

dduvall changed the task status from Stalled to Open.Oct 16 2017, 5:11 PM

Credentials are now handled via a docker-pusher script

The push failing with a 403 is now T178606

As a suggestion: I would host your own registry under the CI project in labs for testing/managing local build you might need to retrieve.

Change 380551 merged by jenkins-bot:
[integration/config@master] Experimental service pipeline jobs