Page MenuHomePhabricator

Develop a CI Testing and Release pipeline for the SCAP package (and potentially other debian packages that we maintain)
Closed, ResolvedPublic

Description

  1. Tag the repo in phabricator with meta-build-package,
    • set up phabricator project
  2. A Herald rule that matches said project tag will trigger a build plan in Harbormaster
    • create herald rule
    • Done: {H134}
  3. Harbormaster tells Jenkins to build a package, via the jenkins-harbormaster plugin
  4. Jenkins builds a test deb based on the specified revision, with the help of jenkins-debian-glue
  5. The jenkins job publishes the package directly to our apt repo on deployment-puppetmaster
    • Use aptly to set up an apt repo on deployment-puppetmaster
    • figure out how to get the package into the apt repo automatically
  6. Need to publish the apt repo so that all beta instances can pull updates from it.
    • Set up HTTP server and configure it to publish the apt repo to other beta instances
    • Set up apt sources entries on each beta instance to pull from this new repo (in addition to the existing sources)
  7. Developer logs into beta cluster and manually upgrades the package, verifies everything is good, etc.
  8. Extra Credit:
    • come up with a way of auto-installing the package into beta, then running automated tests that depend on the package.

Related Objects

View Standalone Graph
This task is connected to more than 200 other tasks. Only direct parents and subtasks are shown here. Use View Standalone Graph to show more of the graph.

Event Timeline

mmodell created this task.Feb 22 2016, 7:50 PM
Restricted Application added subscribers: StudiesWorld, Aklapper. · View Herald TranscriptFeb 22 2016, 7:50 PM
mmodell claimed this task.Feb 22 2016, 7:51 PM
mmodell triaged this task as Normal priority.
mmodell added subscribers: thcipriani, hashar.

I highly recommend http://jenkins-debian-glue.org/ which I have been pet playing for ages. It is a few shell scripts that wrap around git-buildpackage (build tool based on source package hosted in git) and cowbuilder (pbuilder with a copy on write image).

The author, a Debian Developer, pushed it to Debian testing and we should have it on our jessie-wikimedia. There are some Jenkins jobs examples in our integration/config.git repo, look for 'debian-glue'

To revision the build images on a server/instance, have a look at the package_builder puppet module written by @akosiaris .

A first could be to craft a job which poll the Differential repo, build and rsync the result somewhere.

See also T127828: scap3 upstream/debian versioning which is about providing a sane configuration for git build package

So here's the plan:

  1. Tag the repo in phabricator with meta-build-package,
  2. A Herald rule that matches said project tag will trigger a build plan in Harbormaster
  3. Harbormaster tells Jenkins to build a package, via the jenkins-harbormaster plugin
  4. Jenkins builds a test deb based on the specified revision, with the help of jenkins-debian-glue
  5. The jenkins job publishes the package directly to our apt repo on deployment-puppetmaster
  6. Developer logs into beta cluster and manually upgrades the package, verifies everything is good, etc.
  7. Bonus points: we could support installing and running automated tests in beta after the package is built, but for now the package testing will be strictly manual.
mmodell updated the task description. (Show Details)Feb 29 2016, 6:28 PM
mmodell updated the task description. (Show Details)Feb 29 2016, 6:35 PM
mmodell updated the task description. (Show Details)Feb 29 2016, 6:42 PM
mmodell updated the task description. (Show Details)Feb 29 2016, 6:45 PM
Luke081515 added a subscriber: Luke081515.
mmodell moved this task from Needs triage to Debt on the Scap board.Mar 4 2016, 7:09 PM

Ok this works now. The package gets installed in a repository on deployment-puppetmaster, though maybe that wasn't the best choice for where to put it. I haven't set up a web server to publish that repo to other instances in the project, nor configured an apt source entry to point to that. Otherwise this is done.

The nice thing is you don't have to make a new jenkins job (nor harbormaster build plan) in order to make a new test-deb. All you need to do is tag the project's repository with meta-build-package and the rest is handled automatically!

mmodell updated the task description. (Show Details)Mar 6 2016, 2:09 AM
mmodell updated the task description. (Show Details)Mar 6 2016, 2:13 AM
mmodell closed this task as Resolved.Mar 6 2016, 3:31 PM

The nice thing is you don't have to make a new jenkins job (nor harbormaster build plan) in order to make a new test-deb. All you need to do is tag the project's repository with meta-build-package and the rest is handled automatically!

Come again ? You mean that ANY repo can be tagged with meta-build-package and we get a Debian package build pipeline ? If yes, I am thrilled!!!!

Gehel added a subscriber: Gehel.Mar 7 2016, 5:20 PM

The nice thing is you don't have to make a new jenkins job (nor harbormaster build plan) in order to make a new test-deb. All you need to do is tag the project's repository with meta-build-package and the rest is handled automatically!

Come again ? You mean that ANY repo can be tagged with meta-build-package and we get a Debian package build pipeline ? If yes, I am thrilled!!!!

That's exactly what I mean :)

@akosiaris: currently this works with differential revisions only, however, I might be able to make it run for every commit if you want to use it with gerrit - but you'd have to merge in gerrit for phabricator to see the commit and trigger the build.