Page MenuHomePhabricator

PoC bare-metal server allocation in labs -- bootstrap mode
Closed, ResolvedPublic

Description

We know we want to do Ironic but it is a probable longer term goal. For now we are pursuing limited but reusable integration of hardware into labs for end users.


Someday we should adopt the openstack 'Ironic' project to support allocation of bare-metal servers for internal projects. I know that analytics would love to have this option, and there are plenty of other test scenarios where this would save a lot of time dickering over hardware allocation and permissions.

Event Timeline

Andrew raised the priority of this task from to Needs Triage.
Andrew updated the task description. (Show Details)
Andrew added a project: Cloud-Services.
Andrew subscribed.
yuvipanda set Security to None.

An alternative to this would be to provide hardware that's:

  1. In the labs subnet
  2. Runs off the labs puppetmaster
  3. Uses LDAP for credential management
  4. Uses LDAP (or the ENC) for puppet (This allows making them self hosted puppetmasters)

^ Would probably be far simpler to get off the ground than Ironic, but will have to be weighted against the fact that it's somewhat duct-tapey hand-rolled solution.

Problems to be solved:

  1. How will this fit into 'projects' + related networking?
  2. How will allocation work? Do we have a pool that rotates? Or is it the same process that we have right now for requesting new hardware?
  3. How will access control work?
  4. More I'm not thinking of, for sure!

Some big projects (video and mwoffliner, at least) on labs are using NFS simply because they need to deal with datasets a bit too large to fit onto instance storage. provisioning simple machines might be a good alternative here.

If the problem is instance storage, AFAIUI the current sizes are arbitrary and in addition, extra storage volumes can be added manually to individual instances (and existing ones resized?; cf. http://docs.openstack.org/openstack-ops/content/user_facing_block_storage.html).

So IMHO that path should be explored first unless the virtualization swallows so much performance that there is a user-visible significant difference.

However, for projects using Labs as a beta test before moving to production, a bare metal sandbox would certainly be interesting.

I'm going to test this with real hardware, and we can come up with an allocation process for it later.

Things to do to make it know it is labs:

  1. Set 'realm' to labs
  2. Set hostname to something.eqiad.wmflabs
  3. Set instanceproject to something meaningful

Then puppet should do the rest.

the original request says: " would save a lot of time dickering over hardware allocation"

one of the follow-ups is immediately 'How will allocation work? Do we have a pool that rotates?"

that may be interpreted as a contradiction a bit

Another question relevant to services like Revscoring (T106867) is access from production to these semi-production services. Currently, the labs network is not accessible from production, and we might not want to open up all of labs. A separate vlan for semi-production services could potentially provide better isolation.

mwoffliner as it currently is is a good test case, I think. I've poked Kelson on IRC and will update.

chasemp renamed this task from Support bare-metal server allocation in labs to Support bare-metal server allocation in labs -- bootstrap mode.Oct 26 2015, 7:34 PM
chasemp updated the task description. (Show Details)
chasemp renamed this task from Support bare-metal server allocation in labs -- bootstrap mode to PoC bare-metal server allocation in labs -- bootstrap mode.Dec 3 2015, 7:38 PM
chasemp updated the task description. (Show Details)
chasemp removed a project: Labs-Team-Backlog.

Change 257663 had a related patch set uploaded (by Cmjohnson):
Adding dhcp entry for promethium bug: task T95185

https://gerrit.wikimedia.org/r/257663

Change 257663 merged by Cmjohnson:
Adding dhcp entry for promethium bug: task T95185

https://gerrit.wikimedia.org/r/257663

Change 257805 had a related patch set uploaded (by Andrew Bogott):
Allow access to the labs puppetmaster from labs metal hosts.

https://gerrit.wikimedia.org/r/257805

Change 257805 merged by Andrew Bogott:
Allow access to the labs puppetmaster from labs metal hosts.

https://gerrit.wikimedia.org/r/257805

Change 258051 had a related patch set uploaded (by Andrew Bogott):
Allow hiera to override $::labsproject

https://gerrit.wikimedia.org/r/258051

Change 258051 merged by Andrew Bogott:
Allow hiera to override $::labsproject

https://gerrit.wikimedia.org/r/258051