Maniphest T207346

Decide where to store jobs for releases-jenkins
Closed, ResolvedPublic
Actions

Description

We eventually want the releases-jenkins instance to run tests and integrate security patches and automagically cut releases.

Since these tests have a large overlap with our existing CI jenkins we could use the existing integration/config JJB to store job definitions. Alternatively, we could use a new repository using either JJB or the Jenkins Pipeline (either declarative or groovy variants).

Using integration/config

Pros

Lots of edge-cases handled already, tests are vetted ever day due to heavy use
As tests change in ci-jenkins, there will not be any additional work needed to update releases-jenkins tests (although divergence of needs is a con)
Less setup work needed (possibly) since there are already existing tests
Familiarity: everyone involved with the releases-jenkins project has some level of familiarity with integration/config

Cons

Lots of likely unneeded configuration for jobs
I don't anticipate releases-jenkins using zuul, so many of the job definitions will be incompatible
The way we deploy jobs via jenkins-job-builder will need to be scripted to only deploy a subset of jobs/current documentation for jjb deployment will need to be updated
The needs of releases-jenkins may significantly diverge from the needs of ci-jenkins resulting in the need to split repositories eventually

Related Objects

Mentioned In: T208528: Create fab task to deploy jenkins job to either ci-jenkins or releases-jenkins
Mentioned Here: T156445: Streamline/automate MW tarball security release process

Event Timeline

thcipriani created this task.Oct 17 2018, 11:14 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 17 2018, 11:14 PM

mmodell added a subscriber: mmodell.Oct 22 2018, 4:29 PM

I think we should start out using integration/config, and if we hit some blockers, we can split later on. Also we could try using a jjb-releases directory before moving to a separate repo outright.

There are two main needs we have right now for releases-jenkins AIUI

building secret tarballs with security patches
running CI tests against security patches

For the first, I think we can have the same exact job in both releases-jenkins and ci-jenkins. The only difference being that releases-jenkins will apply some git patches first. So the job might do a if /srv/patches/$branch/*.patch exist then apply them, which gets skipped in the ci-jenkins since that directory doesn't exist. Then we could just deploy an identical job to both servers.

For the latter, I think we want to continue using the same exact docker images, but the way we'd get the patches to releases-jenkins would be different (we wouldn't use zuul-merger). I think. I'm not really sure how we'd construct these jobs.

integration/config and a jjb-releases directory seems a good start. To generate the jobs we would just point jenkins-jobs to a different config file and directory. Eg instead of:

tox -e venv -- jenkins-jobs --conf jenkins_jobs.ini update config/jjb

One would use:

tox -e venv -- jenkins-jobs --conf jenkins-releases.ini update config/jjb-releases

Bonus point and your favorite drink when both are harnessed in a Fabric task. That assumes using JJB DSL to generate the job, I don't think it has much support for the Pipeline syntax.

For the Pipeline syntax, potentially the jobs could be in a jenkins-releases directory with the Groovy files in it. But I do not know how they can then be deployed to the Jenkins instance.

To run MediaWiki tests, we can rely on Quibble which has all the logic. Potentially the job would prepare the source code (decompress the tarball, get the dependencies from composer or vendor.git) then Quibble can be instructed to NOT fetch anything by passing it --skip-zuul. Simply mount the prepared source code to /workspace/src.

Side note: Zuul has support for timed pipelines (eg run on a daily basis) and mail reporting, but it is not very convenient. It needs a new pipeline per scheduled periods (one for daily, one for hourly, etc).

sbassett added a subscriber: sbassett.Nov 1 2018, 5:35 PM

Sounds like the general consensus is to start with integration/config and work from there, so I'll got ahead and close out this task.

This task, of course, begets more tasks so those will be coming soon. I'm going to start adding these as subtasks for T156445.

In T207346#4686795, @Legoktm wrote:

I think we should start out using integration/config, and if we hit some blockers, we can split later on. Also we could try using a jjb-releases directory before moving to a separate repo outright.
There are two main needs we have right now for releases-jenkins AIUI

building secret tarballs with security patches

running CI tests against security patches

For the first, I think we can have the same exact job in both releases-jenkins and ci-jenkins. The only difference being that releases-jenkins will apply some git patches first. So the job might do a if /srv/patches/$branch/*.patch exist then apply them, which gets skipped in the ci-jenkins since that directory doesn't exist. Then we could just deploy an identical job to both servers.
For the latter, I think we want to continue using the same exact docker images, but the way we'd get the patches to releases-jenkins would be different (we wouldn't use zuul-merger). I think. I'm not really sure how we'd construct these jobs.

Seems like that is well underway as of this morning: https://gerrit.wikimedia.org/r/#/c/integration/config/+/471036/

In T207346#4691282, @hashar wrote:
integration/config and a jjb-releases directory seems a good start. To generate the jobs we would just point jenkins-jobs to a different config file and directory. Eg instead of:
tox -e venv -- jenkins-jobs --conf jenkins_jobs.ini update config/jjb
One would use:
tox -e venv -- jenkins-jobs --conf jenkins-releases.ini update config/jjb-releases
Bonus point and your favorite drink when both are harnessed in a Fabric task. That assumes using JJB DSL to generate the job, I don't think it has much support for the Pipeline syntax.

This seems like a good idea. I'll make a task for it.

For the Pipeline syntax, potentially the jobs could be in a jenkins-releases directory with the Groovy files in it. But I do not know how they can then be deployed to the Jenkins instance.

I'm not particularly married to using Jenkins pipeline for this. The way we've done groovy for the service-pipeline job within zuul is to use zuul's raw_include which has worked well enough. For tarballs it doesn't sound like we need a pipeline script just yet, so this is probably moot.