As a follow-up to T150501: Spike: Evaluate experimental Docker based CI w/ scap builds we'd like to explore using Kubernetes in a similar fashion to provide containerized build environments for CI.
Preliminary questions
- Should we continue with Docker for this PoC or experiment with Rkt? Ops seems more interested in the latter but we've already had success with Docker in T150501: Spike: Evaluate experimental Docker based CI w/ scap builds and the existing ops/puppet modules are designed to work with it.
- Should this PoC extend to CI infrastructure itself (managing CI slaves) or should we limit it in scope to just build environments? Perhaps we should just start with the build environment and let the result inform what changes we make, if any, to the Jenkins slave pool.
- Should this PoC extend to promotion (saving and registering) of images as artifacts?
Questions we hope to answer with this PoC
- Can we maintain the separation of concern for image manifests (and now extended to cluster manifests) we achieved with T150501: Spike: Evaluate experimental Docker based CI w/ scap builds? Specifically, can we build images and provision a cluster based on manifests from the repo cloned by the Jenkins slave?
- Will Kubernetes provision with adequately low overhead? The overhead of straight Docker based build environments was incredibly low (see T150504#2812971). Can we achieve something comparable with k8s?
- Will k8s provide adequate caching and garbage collection of intermediate image layers and containers?
- Can the k8s manifests be general enough to describe both development and CI (and possibly staging/production) environments?
PoC design (needs related tasks)
- Labs project and a handful (?) of instances for hosting the experimental k8s cluster.
- Application and refactoring, as needed, of existing k8s ops/puppet modules
- Instance in the existing integration project to host experimental builds and orchestrate pods in the cluster, similar to the one created for T150502: Set up experimental Docker CI slave
- Guinea-pig project that will host cluster/image manifests
- scap and malu worked well for this in testing T150501: Spike: Evaluate experimental Docker based CI w/ scap builds)
- Harbormaster build plan for triggering Jenkins builds from submitted Diff revisions
- A generalized Jenkins job, similar to T150504: Define generic job that runs unit tests within a Docker container, for orchestrating the build's k8s cluster and archiving build artifacts