Page MenuHomePhabricator

Decide where to store jobs for releases-jenkins
Closed, ResolvedPublic

Description

We eventually want the releases-jenkins instance to run tests and integrate security patches and automagically cut releases.

Since these tests have a large overlap with our existing CI jenkins we could use the existing integration/config JJB to store job definitions. Alternatively, we could use a new repository using either JJB or the Jenkins Pipeline (either declarative or groovy variants).

Using integration/config

Pros

  1. Lots of edge-cases handled already, tests are vetted ever day due to heavy use
  2. As tests change in ci-jenkins, there will not be any additional work needed to update releases-jenkins tests (although divergence of needs is a con)
  3. Less setup work needed (possibly) since there are already existing tests
  4. Familiarity: everyone involved with the releases-jenkins project has some level of familiarity with integration/config

Cons

  1. Lots of likely unneeded configuration for jobs
  2. I don't anticipate releases-jenkins using zuul, so many of the job definitions will be incompatible
  3. The way we deploy jobs via jenkins-job-builder will need to be scripted to only deploy a subset of jobs/current documentation for jjb deployment will need to be updated
  4. The needs of releases-jenkins may significantly diverge from the needs of ci-jenkins resulting in the need to split repositories eventually

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 17 2018, 11:14 PM

I think we should start out using integration/config, and if we hit some blockers, we can split later on. Also we could try using a jjb-releases directory before moving to a separate repo outright.

There are two main needs we have right now for releases-jenkins AIUI

  • building secret tarballs with security patches
  • running CI tests against security patches

For the first, I think we can have the same exact job in both releases-jenkins and ci-jenkins. The only difference being that releases-jenkins will apply some git patches first. So the job might do a if /srv/patches/$branch/*.patch exist then apply them, which gets skipped in the ci-jenkins since that directory doesn't exist. Then we could just deploy an identical job to both servers.

For the latter, I think we want to continue using the same exact docker images, but the way we'd get the patches to releases-jenkins would be different (we wouldn't use zuul-merger). I think. I'm not really sure how we'd construct these jobs.

hashar added a subscriber: hashar.Oct 24 2018, 9:53 AM

integration/config and a jjb-releases directory seems a good start. To generate the jobs we would just point jenkins-jobs to a different config file and directory. Eg instead of:

tox -e venv -- jenkins-jobs --conf jenkins_jobs.ini update config/jjb

One would use:

tox -e venv -- jenkins-jobs --conf jenkins-releases.ini update config/jjb-releases

Bonus point and your favorite drink when both are harnessed in a Fabric task. That assumes using JJB DSL to generate the job, I don't think it has much support for the Pipeline syntax.

For the Pipeline syntax, potentially the jobs could be in a jenkins-releases directory with the Groovy files in it. But I do not know how they can then be deployed to the Jenkins instance.

To run MediaWiki tests, we can rely on Quibble which has all the logic. Potentially the job would prepare the source code (decompress the tarball, get the dependencies from composer or vendor.git) then Quibble can be instructed to NOT fetch anything by passing it --skip-zuul. Simply mount the prepared source code to /workspace/src.

Side note: Zuul has support for timed pipelines (eg run on a daily basis) and mail reporting, but it is not very convenient. It needs a new pipeline per scheduled periods (one for daily, one for hourly, etc).

thcipriani closed this task as Resolved.Nov 1 2018, 5:55 PM
thcipriani claimed this task.

Sounds like the general consensus is to start with integration/config and work from there, so I'll got ahead and close out this task.

This task, of course, begets more tasks so those will be coming soon. I'm going to start adding these as subtasks for T156445.

I think we should start out using integration/config, and if we hit some blockers, we can split later on. Also we could try using a jjb-releases directory before moving to a separate repo outright.

There are two main needs we have right now for releases-jenkins AIUI

  • building secret tarballs with security patches
  • running CI tests against security patches

For the first, I think we can have the same exact job in both releases-jenkins and ci-jenkins. The only difference being that releases-jenkins will apply some git patches first. So the job might do a if /srv/patches/$branch/*.patch exist then apply them, which gets skipped in the ci-jenkins since that directory doesn't exist. Then we could just deploy an identical job to both servers.

For the latter, I think we want to continue using the same exact docker images, but the way we'd get the patches to releases-jenkins would be different (we wouldn't use zuul-merger). I think. I'm not really sure how we'd construct these jobs.

Seems like that is well underway as of this morning: https://gerrit.wikimedia.org/r/#/c/integration/config/+/471036/

integration/config and a jjb-releases directory seems a good start. To generate the jobs we would just point jenkins-jobs to a different config file and directory. Eg instead of:

tox -e venv -- jenkins-jobs --conf jenkins_jobs.ini update config/jjb

One would use:

tox -e venv -- jenkins-jobs --conf jenkins-releases.ini update config/jjb-releases

Bonus point and your favorite drink when both are harnessed in a Fabric task. That assumes using JJB DSL to generate the job, I don't think it has much support for the Pipeline syntax.

This seems like a good idea. I'll make a task for it.

For the Pipeline syntax, potentially the jobs could be in a jenkins-releases directory with the Groovy files in it. But I do not know how they can then be deployed to the Jenkins instance.

I'm not particularly married to using Jenkins pipeline for this. The way we've done groovy for the service-pipeline job within zuul is to use zuul's raw_include which has worked well enough. For tarballs it doesn't sound like we need a pipeline script just yet, so this is probably moot.

To run MediaWiki tests, we can rely on Quibble which has all the logic. Potentially the job would prepare the source code (decompress the tarball, get the dependencies from composer or vendor.git) then Quibble can be instructed to NOT fetch anything by passing it --skip-zuul. Simply mount the prepared source code to /workspace/src.

One note: releases-jenkins doesn't currently have docker installed on it, but it sounds like we'll need docker to run images there if the two instances are going to share jobs.