This is a brain dump of aough plan to migrate out of Nodepool. Saved in Phabricator to make it easier to find

Rough plan to migrate out of Nodepool

The challenge is to provision one or more Docker images suitable for CI.
The Nodepool image contains all the dependencies required although it is rather large (~4GB)

Ideas:

Use the same image as Nodepool

We do the Nodepool image provisionning with the diskimage-builder utility (DIB). It produces a qcow2 image that is then uploaded to OpenStack.

1. DIB supports generating and pushing the image to a Docker registry
2. Alternatively we can export the qcow2 image to tar and use 'docker import':

$ virt-tar-out -a image-jessie-20170901T201648Z.qcow2 / - | docker import - qcow2import
eb2d6e46ec2156d28312bf1988a426a4b6e003a2f85b8d06657b6ce0407b4281
$ docker images
REPOSITORY          TAG                 IMAGE ID            CREATED              VIRTUAL SIZE
qcow2import         latest              eb2d6e46ec21        About a minute ago   4.273 GB
$

The end result is a 4.275GBytes once imported in Docker. The CI hosts currently use the public Docker hub and I am not sure how much disk space that will end up consuming.

Port Jenkins jobs

Next challenge would be how to rewrite the jobs so that instead of using node: ci-jessie-wikimedia they use Docker. There is a Docker plugin to help in that, but @dduvall found out it was causing some issues or added little value (T150505).

There are FOUR plugins in Jenkins:

1. Docker Plugin

A property

https://docs.openstack.org/infra/jenkins-job-builder/properties.html?highlight=docker#properties.docker-container

https://wiki.jenkins.io/display/JENKINS/Docker+Plugin

It spans containers on demand, attaches them to master, like Nodepool). Probably the easiest to integrate/deploy.

The aim of the docker plugin is to be able to use a docker host to dynamically provision a slave, run a single build, then tear-down that slave.

UNTESTED

- node: ci-jessie-wikimedia
+ node: DebianJessieDocker
+ wrappers:
+     - docker-custom-build-env:
+            image-type: pull
+            image: wmfreleng/ci-jessie-wikimedia
 ...

From there we can spin off more docker hosts as we migrated the jobs as is from ci-jessie-wikimedia Nodepool images to Docker containers.

2. Wrapper (from CloudBees)

https://docs.openstack.org/infra/jenkins-job-builder/wrappers.html?highlight=docker#wrappers.docker-custom-build-env
https://wiki.jenkins.io/display/JENKINS/CloudBees+Docker+Custom+Build+Environment+Plugin

Orphan/legacy

3. A builder

https://docs.openstack.org/infra/jenkins-job-builder/builders.html?highlight=docker#builders.docker-build-publish
https://wiki.jenkins.io/display/JENKINS/Docker+build+publish+Plugin

Probably legacy / dupe of CloudBees above)

4. Docker pipeline

https://wiki.jenkins.io/display/JENKINS/Docker+Pipeline+Plugin
https://jenkins.io/doc/book/pipeline/docker/

Would require to rewrite all the JJB jobs to use the pipeline syntax.

Porting to the pipeline syntax should be long term goal.

Conclusion

Short term: "Docker Plugin" which might be easy to integrate and would be back compatibile with our job definitions.

Mid term: rewrite jobs to use Pipeline syntax.

Long term: infra to a K8S

Food for later

Most probably trusty is out of scope. It is almost phased out. Pending migration of php 5.5 jobs to Jessie (WIP)

Later we can look at provisionning customer Docker images per project. Potentially using blubber but it is not ready yet.