Define new Jenkins pipeline for container build phase
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	dduvall
	Sep 7 2017, 6:22 PM

Project Tags

Referenced Files

None

Subscribers

Description

We've sketched out a rough outline for the build phase of the container release pipeline. We should implement this as a Jenkins pipeline (as in the Jenkins Pipeline plugin) and see if we can't get something suitable for building Mathoid images.

blubber build test image
docker run test image entrypoint
decision/feedback fork:
- test entrypoint passes
  - blubber build production image
  - push image to WMF docker registry
  - provide feedback
- test entrypoint fails
  - abort pipeline
  - provide feedback

Details

Related Changes in Gerrit:

	Subject	Author	Repo	Branch	Lines +/-
	Experimental service pipeline jobs	Dduvall	integration/config	master	+101 -0

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Resolved	None	T170453 FY2017/18 Program 6: Streamlined Service delivery
Invalid	None	T170480 FY2017/18 Program 6 - Outcome 2: Developers are able to develop and test their applications through a unified pipeline towards production deployment.
Invalid	None	T170481 FY2017/18 Program 6 - Outcome 2 - Objective 2: Set up a continuous integration and deployment pipeline
Resolved	None	T157469 Mathoid CI Container Build
Resolved	dduvall	T175297 Define new Jenkins pipeline for container build phase
Resolved	thcipriani	T175293 Provision Docker >= 17.05 on contint1001
Resolved	dduvall	T175296 Install Blubber on contint1001
Resolved	dduvall	T175609 Package Blubber
Resolved	thcipriani	T173128 Find CI container build location
		Restricted Task
Resolved	dduvall	T176896 Establish secure way of passing registry credentials from Jenkins to Docker
Resolved	thcipriani	T175298 Secret storage on contint1001 for Docker registry password
Resolved	Joe	T178606 Auth fails for `docker-pusher` script on `contint1001`

Event Timeline

dduvall created this task.Sep 7 2017, 6:22 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 7 2017, 6:22 PM

dduvall added a parent task: T157469: Mathoid CI Container Build.Sep 7 2017, 6:23 PM

dduvall added a subtask: T175293: Provision Docker >= 17.05 on contint1001.

dduvall added a subtask: T175296: Install Blubber on contint1001.

dduvall added a subtask: T175298: Secret storage on contint1001 for Docker registry password.Sep 7 2017, 6:40 PM

dduvall added a subtask: T173128: Find CI container build location.

akosiaris closed subtask T175293: Provision Docker >= 17.05 on contint1001 as Resolved.Sep 21 2017, 2:28 PM

Change 380551 had a related patch set uploaded (by Dduvall; owner: Dduvall):
[integration/config@master] WIP Service pipeline DSL

https://gerrit.wikimedia.org/r/380551

gerritbot added a project: Patch-For-Review.Sep 26 2017, 5:21 PM

dduvall claimed this task.Sep 26 2017, 5:22 PM

dduvall triaged this task as Medium priority.

dduvall moved this task from Backlog to CI on the Release Pipeline board.

Restricted Application edited projects, added: Release-Engineering-Team (Kanban); removed: Release-Engineering-Team (Next). · View Herald TranscriptSep 26 2017, 5:22 PM

dduvall moved this task from Backlog to In-progress on the Release-Engineering-Team (Kanban) board.Sep 26 2017, 5:22 PM

An experimental job has been created from the current JJB patchset and is working up until just before the registry push.

Successfully built 9d25c234a7aa
Successfully tagged mediawiki-services-mathoid:build-20
[Pipeline] dockerFingerprintFrom
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
java.io.IOException: Cannot retrieve .Id from 'docker inspectdocker-registry.wikimedia.org/nodejs-devel AS prep'
	at org.jenkinsci.plugins.docker.workflow.client.DockerClient.inspectRequiredField(DockerClient.java:203)
	at org.jenkinsci.plugins.docker.workflow.FromFingerprintStep$Execution.run(FromFingerprintStep.java:119)
	at org.jenkinsci.plugins.docker.workflow.FromFingerprintStep$Execution.run(FromFingerprintStep.java:75)
	at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1$1.call(AbstractSynchronousNonBlockingStepExecution.java:47)
	at hudson.security.ACL.impersonate(ACL.java:260)
	at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1.run(AbstractSynchronousNonBlockingStepExecution.java:44)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:473)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1152)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:622)
	at java.lang.Thread.run(Thread.java:748)
Finished: FAILURE

This failure may be due to an unresolved bug.

The more specifically relevant bug may be this one which indicates it's an issue with the FROM aliases used for multistage. The recommended workaround is to simply use sh to execute docker build for now. :) Easy enough.

Patchset 4 successfully builds the production image but fails with a 403 when trying to push to the registry. It's unclear whether our uploader credential is being used.

On integration-slave-docker-1705 I changed the docker.service to use debug log level (in ExecStart, pass -D) and journalctl -u docker --follow` to watch.

Then docker push docker-registry.wikimedia.org/wikimedia/mediawiki-services-mathoid:build-28 and in the system log:

msg="Calling GET /_ping"
msg="Calling POST /v1.30/images/docker-registry.wikimedia.org/wikimedia/mediawiki-services-mathoid/push?tag=build-28"
msg="hostDir: /etc/docker/certs.d/docker-registry.wikimedia.org"
msg="Trying to push docker-registry.wikimedia.org/wikimedia/mediawiki-services-mathoid to https://docker-registry.wikimedia.org v2"
msg="Pushing repository: docker-registry.wikimedia.org/wikimedia/mediawiki-services-mathoid:build-28"

Then there is a 403.

Using docker login and the credential works all fine though:

$ docker login docker-registry.wikimedia.org
Username (uploader): uploader
Password
Login Succeeded
$

Who knows whether docker POST with the credentials or if we get access for it :(

Clarified with @Joe labs instances are not allowed to interact with the docker registry. Nginx rejects them and that is based on the puppet bits:

modules/profile/manifests/docker/registry.pp

class profile::docker::registry {
    $image_builders = hiera(
        'profile::docker::registry::image_builders',
        $network::constants::special_hosts[$::realm]['deployment_hosts']
    )
...
    class { '::docker::registry::web':
        allow_push_from      => $image_builders,
...

And in hiera:

hieradata/role/common/docker/registry.yaml

profile::docker::registry::image_builders: ['10.64.16.176'] # copper.eqiad.wmnet

Hence the 403 is working as intended.

So I guess that has to be build on a production slave, potentially with a dedicated Jenkins instance?

Right on. Thanks for debugging that @hashar! I think we can continue to test the experimental job on the labs instance and just push to Docker Hub for now. Once we get the dependencies installed on contint1001 (blubber, docker >= 17.05) and the job is somewhat stable, we can move it there.

Another issue that has arisen in testing this is that the default credential store for Docker is very insecure. By default, it stores all usernames/passwords given to docker login in the clear (in ~/.docker/config.json) until a subsequent docker logout is called. That means there is a substantial window of time where any other Jenkins job running via the same remote agent could gain access to the registry credentials. I'll open a subtask to further discuss and resolve this issue.

dduvall mentioned this in T176896: Establish secure way of passing registry credentials from Jenkins to Docker.Sep 27 2017, 6:18 PM

dduvall added a subtask: T176896: Establish secure way of passing registry credentials from Jenkins to Docker.

dduvall removed a subtask: T175298: Secret storage on contint1001 for Docker registry password.Oct 2 2017, 7:48 PM

hashar closed subtask T175296: Install Blubber on contint1001 as Resolved.Oct 4 2017, 8:59 AM

greg moved this task from In-progress to Blocked (externally) on the Release-Engineering-Team (Kanban) board.Oct 11 2017, 4:54 PM

dduvall mentioned this in T178022: Include Blubber metadata in Dockerfile output as labels.Oct 11 2017, 11:51 PM

thcipriani closed subtask T176896: Establish secure way of passing registry credentials from Jenkins to Docker as Resolved.Oct 16 2017, 5:09 PM

thcipriani closed subtask T173128: Find CI container build location as Resolved.

dduvall changed the task status from Stalled to Open.Oct 16 2017, 5:11 PM

dduvall moved this task from Blocked (externally) to In-progress on the Release-Engineering-Team (Kanban) board.