Page MenuHomePhabricator

[EPIC] Run CI jobs in disposable VMs
Closed, ResolvedPublic

Description

We need to run unit tests under Jenkins whenever someone submit a new patchset regardless of the submitter status (ie a potential attacker). To achieve that, the Jenkins job need to be properly isolated.

Following an internal WMF meeting, we could setup virtual machines using vagrant. They would be booted on a second box and controlled by Jenkins vagrant plugin.


Note: this tracking task comes from Bugzilla, although it still serves its purpose, most of the activity is tracked on the workboard for .Continuous-Integration-Scaling

Details

Reference
bz45499

Related Objects

StatusAssignedTask
Resolvedhashar
OpenNone
InvalidRyasmeen
DeclinedNone
DeclinedNone
Resolvedhashar
Resolvedhashar
Resolvedhashar
Resolvedhashar
Resolvedhashar
DuplicateNone
Resolved bd808
DuplicateNone
DeclinedNone
Declinedhashar
Resolvedhashar
DeclinedNone
Declinedchasemp
DeclinedNone
Resolvedcoren
Invalidhashar
Resolvedhashar
Declinedhashar
ResolvedCmjohnson
ResolvedCmjohnson
Resolvedhashar
ResolvedKrinkle
ResolvedKrinkle
ResolvedKrinkle
Resolvedhashar
ResolvedKrinkle
Resolvedhashar
Resolvedhashar
Resolvedhashar
Resolvedhashar
Resolvedhashar
ResolvedAndrew
ResolvedKrinkle
Resolvedhashar
ResolvedKrinkle
ResolvedPaladox
ResolvedLegoktm
ResolvedLegoktm
ResolvedLegoktm
ResolvedLegoktm
ResolvedPaladox
ResolvedLegoktm
ResolvedLegoktm
Resolvedhashar
Resolvedhashar
Resolved yuvipanda

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
faidon added a subscriber: faidon.Dec 3 2014, 1:45 PM

(In reply to comment #13)

I experimented in the past with minijail0, from the Chromium/Chrome project.
It might be worthwhile to experiment some more, it will certainly be much much
faster than anything VM-related.

The documentation is sparse :( I am not sure how it manage to isolate a process, apparently it works using a chroot which I am not sure how we can set it up :(

There's a new kid on the block that seems to do similar things to minijail0 (seccomp etc.) but which seems far easier to use: https://l3net.wordpress.com/projects/firejail/ -- maybe worth a look. I haven't audited from a security perspective, so I'm definitely not recommending it just yet, just throwing it out there in case you want to consider it.

demon removed a subscriber: demon.Dec 3 2014, 6:04 PM
greg added a comment.Dec 20 2014, 12:47 AM

(Just came across this, pasting for record keeping: https://wiki.jenkins-ci.org/display/JENKINS/JClouds+Plugin )

hashar added a comment.Jan 6 2015, 5:09 PM
In T47499#937396, @greg wrote:

(Just came across this, pasting for record keeping: https://wiki.jenkins-ci.org/display/JENKINS/JClouds+Plugin )

I am not sure how fast an instance will boot or whether the plugin is able to maintain a pool of VM to consume. Worth looking at though. Filled T85933 to evaluate it.

Originally we were hinting towards using our OpenStack cloud for this. That seems to me still like the most feasible path forwards, because:

  • We don't have to ensure that it is secure and isolated sufficiently. Wikimedia Labs is already responsible for that. If other backends are better, they can evaluate that accordingly.
  • We know it scales.
  • It is already in-place for the most part. (Less set up overhead.)
  • We don't require operations resources we don't have to maintain it. If we were to use some other system, it will likely end up being created by someone who's allocating his/her resources on-off and isn't able to maintain it. Wikimedia Labs has maintainers (not many, but it has infrastructure and will naturally grow as needed).

As for the infrastructure for our pool of slaves specifically, that isn't part of plain OpenStack Cloud by default, however OpenStack maintains a package called nodepool that does this. And they're dedicated to maintaining it as they use it themselves for their Jenkins jobs.

More information: http://ci.openstack.org/nodepool/

Conversation with maintainers about how it works:
https://gist.githubusercontent.com/Krinkle/5aad511288e1515d01c7/raw

Yup definitely heading toward reproducing the OpenStack CI infrastructure which is supported/developped by more than a handful of person.

I found out today that the blocking tasks are rather messy, I talked to Greg about it and I would rewrite / split some of them.

greg set Security to None.Jan 7 2015, 10:31 PM
greg added a subscriber: mmodell.
greg assigned this task to hashar.Feb 24 2015, 1:07 AM
greg renamed this task from Jenkins: Run jobs in disposable VMs to [Quarterly Success Metric] Jenkins: Run jobs in disposable VMs.Feb 26 2015, 10:18 PM
Eloquence moved this task from March 2015: Platform to Tag on the Epic board.Mar 10 2015, 5:43 AM
greg renamed this task from [Quarterly Success Metric] Jenkins: Run jobs in disposable VMs to [keyresult] Run CI jobs in disposable VMs.Aug 22 2015, 12:49 AM
greg updated the task description. (Show Details)
greg renamed this task from [keyresult] Run CI jobs in disposable VMs to [EPIC] Run CI jobs in disposable VMs.Sep 2 2015, 4:18 PM
greg removed a project: releng-201516-q2.
hashar updated the task description. (Show Details)Sep 19 2015, 10:30 AM
Krinkle removed a subscriber: Krinkle.Jun 7 2016, 6:13 PM
hashar closed this task as Resolved.Nov 4 2016, 9:15 AM

I am closing this task, it is no more used and the progress is better tracked in Continuous-Integration-Scaling