Page MenuHomePhabricator

Investigate using Drydock for CI
Closed, DeclinedPublic

Description

drydock is a phabricator prototype application which is designed for allicating and caching resources like VMs, Working Copies, etc and then handing them to harbormaster for executing test jobs.

The upstream phabricator project has been actively developing drydock in recent weeks, for example see changelog/2015.41 and the linked tracking tasks.

Summary: Upstream is self-hosting phabricator builds in harbormaster and building working copies in drydock. It supports multiple trust-groups for isolation and you can assign VM instances to any arbitrary group, then assign tests to a group and they will be executed on the VM instances from the corresponding trust group. The upstream description is probably better than my summary so I encourage you to read more about it there.

This task is just here to remind & encourage us to evaluate these tools and apply them to our own CI needs as much as possible since that will lead to much more integrated infrastructure and hopefully reduced fragmentation of our tooling/expertise.

Event Timeline

mmodell raised the priority of this task from to Medium.
mmodell updated the task description. (Show Details)

How would drydock solve T112560: [tracking] Disposable VMs need a cache for package managers from first impression it seems to be unrelated (it seems more like a nodepool replacement)?

@JanZerebecki: The documentation is sparse but it's a lot more than just a resource pool, at least it is when combined with harbormaster. Also, I believe it could be made to work with nodepool rather than replace it, though I could be wrong.

How would drydock solve T112560: [tracking] Disposable VMs need a cache for package managers from first impression it seems to be unrelated (it seems more like a nodepool replacement)?

For example: "Allow Harbormaster to lease working copies from Drydock"

That seems to mean it is able to run some build steps like checkout of a working copy before other build steps. But the parent task is not about allowing to declare those steps but about which specific steps to use. Which steps would drydoc use to cache remotely obtained dependencies or intermediate build artifacts?

hashar claimed this task.

I went with a lame central rsync server.

Did a first pass using a cache store/restore system based on rsync. Investigated as part of T116017