Page MenuHomePhabricator

[EPIC] Run CI jobs in disposable VMs
Closed, ResolvedPublic

Description

We need to run unit tests under Jenkins whenever someone submit a new patchset regardless of the submitter status (ie a potential attacker). To achieve that, the Jenkins job need to be properly isolated.

Following an internal WMF meeting, we could setup virtual machines using vagrant. They would be booted on a second box and controlled by Jenkins vagrant plugin.


Note: this tracking task comes from Bugzilla, although it still serves its purpose, most of the activity is tracked on the workboard for .Continuous-Integration-Scaling

Details

Reference
bz45499

Related Objects

StatusSubtypeAssignedTask
Resolvedhashar
Resolvedhashar
InvalidRyasmeen
DeclinedNone
DeclinedNone
Resolvedhashar
Resolvedhashar
Resolvedhashar
Resolvedhashar
Resolvedhashar
DuplicateNone
Resolvedbd808
DuplicateNone
DeclinedNone
Declinedhashar
Resolvedhashar
DeclinedNone
Declined chasemp
DeclinedNone
Resolvedcoren
Invalidhashar
Resolvedhashar
Declinedhashar
Resolved Cmjohnson
Resolved Cmjohnson
Resolvedhashar
ResolvedKrinkle
ResolvedKrinkle
ResolvedKrinkle
Resolvedhashar
ResolvedKrinkle
Resolvedhashar
Resolvedhashar
Resolvedhashar
Resolvedhashar
Resolvedhashar
ResolvedAndrew
ResolvedKrinkle
Resolvedhashar
ResolvedKrinkle
ResolvedPaladox
ResolvedLegoktm
ResolvedLegoktm
ResolvedLegoktm
ResolvedLegoktm
ResolvedPaladox
ResolvedLegoktm
ResolvedLegoktm
Resolvedhashar
Resolvedhashar
Resolvedyuvipanda

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
In T47499#937396, @greg wrote:

(Just came across this, pasting for record keeping: https://wiki.jenkins-ci.org/display/JENKINS/JClouds+Plugin )

I am not sure how fast an instance will boot or whether the plugin is able to maintain a pool of VM to consume. Worth looking at though. Filled T85933 to evaluate it.

Originally we were hinting towards using our OpenStack cloud for this. That seems to me still like the most feasible path forwards, because:

  • We don't have to ensure that it is secure and isolated sufficiently. Wikimedia Labs is already responsible for that. If other backends are better, they can evaluate that accordingly.
  • We know it scales.
  • It is already in-place for the most part. (Less set up overhead.)
  • We don't require operations resources we don't have to maintain it. If we were to use some other system, it will likely end up being created by someone who's allocating his/her resources on-off and isn't able to maintain it. Wikimedia Labs has maintainers (not many, but it has infrastructure and will naturally grow as needed).

As for the infrastructure for our pool of slaves specifically, that isn't part of plain OpenStack Cloud by default, however OpenStack maintains a package called nodepool that does this. And they're dedicated to maintaining it as they use it themselves for their Jenkins jobs.

More information: http://ci.openstack.org/nodepool/

Conversation with maintainers about how it works:
https://gist.githubusercontent.com/Krinkle/5aad511288e1515d01c7/raw

Yup definitely heading toward reproducing the OpenStack CI infrastructure which is supported/developped by more than a handful of person.

I found out today that the blocking tasks are rather messy, I talked to Greg about it and I would rewrite / split some of them.

greg added a subscriber: mmodell.
greg renamed this task from Jenkins: Run jobs in disposable VMs to [Quarterly Success Metric] Jenkins: Run jobs in disposable VMs.Feb 26 2015, 10:18 PM
greg renamed this task from [Quarterly Success Metric] Jenkins: Run jobs in disposable VMs to [keyresult] Run CI jobs in disposable VMs.Aug 22 2015, 12:49 AM
greg updated the task description. (Show Details)
greg renamed this task from [keyresult] Run CI jobs in disposable VMs to [EPIC] Run CI jobs in disposable VMs.Sep 2 2015, 4:18 PM
greg removed a project: releng-201516-q2.

I am closing this task, it is no more used and the progress is better tracked in Continuous-Integration-Scaling