Page MenuHomePhabricator

Create labs project for continuous integration nodepool
Closed, DuplicatePublic

Description

Following a discussion I had with Ryan Lane, the continuous integration project could use an OpenStack projects in labs that would be dedicated to running disposable Jenkins instances (bug 45499).

What we would need to have:

  • have the instances fully isolated from the rest of the network (prod / other labs projects), with the exception of communications with Jenkins master and git fetch from Zuul (matrix to be determined). See T86168
  • a write access to the OpenStack API to be able to spawn instances programmatically. See T86170
  • possibly have the project running on dedicated hardware to avoid being influenced by other volunteers projects or to avoid influencing them. See T84989

Version: unspecified
Severity: normal

Details

Reference
bz53978

Event Timeline

bzimport raised the priority of this task from to Needs Triage.Nov 22 2014, 2:05 AM
bzimport set Reference to bz53978.
bzimport added a subscriber: Unknown Object (MLST).

A related post I sent to the labs mailing list
http://lists.wikimedia.org/pipermail/labs-l/2013-November/001796.html

-----------------8<---------------------8<--------------------8<--------

I could use a new labs project on the wmflabs project to boot up new instances that would be consumed by the Continuous integration system. The aim is to be able to properly isolate some tests we are running.

The idea is to maintain a pool of instances that would be dynamically registered as Jenkins slaves of the CI Jenkins master. The later would then be able to send test to run on those slaves and we would destroy the instance once the job is completed.

The requisites would be:

  • isolate the instances from the rest of the prod/labs networks using a yet to be written security matrix. The instances would at least need:
    • to be reachable from the Jenkins master by ssh
    • the availability to fetch from npm
  • A virtual image using Ubuntu Precise and prepopulated withrole::ci::slave::labs::common and contint::slave-scripts. The idea is to have the new instances booting up as fast as possible.
  • A host profile with 2GB of RAM, single CPU and 5GB of /dev/vdb disk.

Optionally:

  • having the project to run on dedicated hardware, but that can happen later on.

Who shall I sync with to make it happen? :-]

Should I fill in a bunch of bugs in Bugzilla for ease of tracking?

-----------------8<---------------------8<--------------------8<--------

I guess I will fill above tasks as bugs depending on this one.

scfc added a comment.Nov 13 2013, 5:30 PM

As this setup sounds *very* similar to the concept of Travis CI (https://travis-ci.org/), have you looked into that? It might be easier than to hammer Labs and Jenkins into shape, and WMF could just sponsor some hardware (or money :-)).

That sounds like a great idea!
As far as I am aware the only functionality travis doesn't currently have that we need would be gerrit integration.

We don't use hosted solutions when we can sanely host them ourselves and we don't use proprietary things unless there's no available alternative.

Setting up a labs project for this is doable, so we should be doing it. We just need to take the time to do it. I think priority wise, this should be after the Labs move to eqiad.

What Ryan said, we can't depend on a third party at all. That doesn't prevent us from reusing Travis open source code and integrate it in our build system, that would nicely scale out CI to more people.

Re Ryan and Hashar, https://github.com/travis-ci, Sorry that is what I meant!
Set up our own slightly modified travis!
I can see our CI taking a big step forward when this bug is resolved :D

We have labs instance in the integration projects: integration-slave1001.eqiad.wmflabs and integration-slave1002.eqiad.wmflabs.

They are fully puppetized and let us fetch packages from nodejs/rubygems/pip...

Still does not provide proper isolation nor the ability to spawn new instance on demand.

Krinkle renamed this task from setup labs project for continuous integration jobs to Create labs project for continuous integration nodepool.Jan 8 2015, 1:17 PM
Krinkle triaged this task as Normal priority.
Krinkle set Security to None.
Krinkle removed a subscriber: Unknown Object (MLST).

I am merging this up in its parent task T47499: [EPIC] Run CI jobs in disposable VMs. Will fill some new tasks for each of the item listed there.