Page MenuHomePhabricator

Install Kubernetes Continuous Deployment Pipeline plugin on ci-staging-jenkins
Closed, ResolvedPublic

Description

In order to evaluate this plugin, we've got to build and install it on ci-staging-jenkins. Since it's not in the available plugins, it must be built and installed manually.

Event Timeline

thcipriani closed this task as Resolved.Mar 11 2017, 11:58 PM

This plugin is installed and has all the jobs that are created in the Pearson base docker image (but not in a docker image). The seed job is looking at a scap repo that includes an environment.bitesize, application.bitesize, and a build.bitesize added. The plugin created a seed job along with the other jobs you see here:

http://ci-staging-jenkins.wmflabs.org/ci/

This is not hooked to k8s or Docker, but it's setup and ready to add those things.

I documented the experience and all the code I used to set this up as a github repo: https://github.com/thcipriani/deployment-pipeline-jenkins-plugin-setup

Some key points in the tl;dr version of how this plugin works:

  1. Every repo has its own Jenkins on k8s
  2. The Jenkins init groovy scripts build a seed job on that Jenkins
  3. That job builds other jobs
  4. All these jobs are written in groovy jobs dsl
  5. These jobs written in groovy dsl build docker images
  6. There is a job to deploy that docker image to k8s

Some random observations from figuring out how this plugin works (kinda sorta):

  1. We have a lot of repos. A Jenkins for every repo is a lot of Jenkinses.
  2. This plugin seems heavily architected in an enterprise sort of way—not necessarily a bad thing but it smells like Java.
  3. Downloads random stuff from the internet, puts it in a Jenkins instance, Jenkins builds containers, Jenkins deploys those containers to some k8s infra — I feel like there's not enough oversight here. If someone replaced the JENKINS_UC_DOWNLOAD envvar with a link to a malicious site then...they could probably make bad things happen (at scale!).

Here is the long(er) version of how this plugin works:

How this works, maybe, for Pearson

In my mind's eye I see a close up of a marble rolling down a track, queue
Powerhouse, the marble drops through a
hole in the track and there is a sound something like a clang of a pan. There
is a boot on a stick that is released, kicking a bowling ball down a track...
A new Jenkins is built via kubectl create. K8s is passed a file that contains
a bunch of environment vars for the underlying image: The git repo for the
project, an SSH private key to access the repo, k8s creds for deployment info,
etc.
Inside the docker image, jenkins.sh is run. It runs through plugins.txt
line-by-line grabbing the versions of plugins specified and pinning them:

curl -sSL [yandex-link] /var/lib/jenkins/plugins/[plugin].jpi
touch /var/lib/jenkins/plugins/[plugin].jpi.pinned

Then Jenkins is launched.
Some init.groovy.d scripts spring into action. The init.groovy.d scripts
create a "seed" job (if one doesn't exist). This job runs everything in the
jobsdsl folder on the new jenkins install.
For reference I exhumed the jobdsl files using:

docker cp <container-id>:/usr/share/jenkins/[whatever] [whatever]

The jobdsl files are groovy scripts that contain large HEREDOC'd groovy
scripts that themselves create new jobs.