@thcipriani and I discussed this on the deployment triage call today.

We resolved to explore the idea of storing the deployment state entirely in the git repo, probably in a branch. This can eliminate the apache dependency and simultaneously resolve T113072: Make puppet provider for scap3, allowing us to get the initial state correct on newly provisioned servers.

Simply assuming the scap/ directory is contained in the deploy repository should be enough. Currently, the services' deploy repositories have the following outline:

deploy
  |-- src/
  |-- node_modules/
  +-- package.json -> src/package.json

So, including scap/ on the same (root) level, would get the config and other stuff inside it automatically available on the targets. As a bonus, that would mean that any changes to the scap/ directory would make for a new commit, meaning that Scap3 would need to track only one hash, not two (one for the repo, the other for the config).

So the idea would be, instead of using the .git/DEPLOY_HEAD file on tin as the canonical latest-deployment for a particular repo, we would simply make a commit to an orphan branch that is pushed upstream as part of a deploy. This file could be used by deploy-local as configuration when fetching from the deploy_host as well as by the future scap puppet provider.

Creating an orphan branch in which to track deployment state would have the benefits:

1. Less intrusive than committing to a repo's deploy branch: no imposing directory structure which may cause collisions.
2. Keeps commit history for a branch clean, automatic changes from deploy tooling wouldn't end-up in the deploy repository.
3. Keeps commit history for the orphan branch clean: should be easy to look through a history of deployments by checking out the orphan branch, rather than having to sort through un-related development history.
4. Keeps the tooling agnostic about how a repo should include the scap directory inside the repository: as a subtree, as a submodule, as a .gitignore'd directory, scap/scap.cfg just has to exist.

An orphan branch also is fairly VCS agnostic (unlike git-tag or git-notes).

That was my thinking in the discussion we had this morning.

From meeting notes 2015-11-30:

One thing of note, is that for production, it is important to use tin for the git_server from which information is fetched and we should enforce that in scap code. tin has much less surface area than using anything and/or everything as an available upstream.

For smaller instances, and to support re-use in labs projects, having a configurable upstream to which we could push a branch with canonical configuration information would get rid of the dependency of apache.

I'm thinking we could start a SimpleHTTPServer rooted at /srv/deployment.

@thcipriani: Is there any way to override the http port that deploy-local uses to connect back to the deployment host?

@mmodell you could include it as part of the git_server argument, e.g., git_server = tin.eqiad.wmflabs:8080.

What changed to point you at evaluating SimpleHTTPServer? Initially I remember some discussion about how that wasn't the right tool for this particular task.

@thcipriani: We were talking about completely replacing the apache server on tin / mirra with a python http implementation. For that task, SimpleHTTPServer almost certainly would not scale. On the other hand, now that we are sure that we need to keep the apache server in order to support the puppet provider, it seems a lot easier to just use a simple http server for labs and one-off deployments from a developer's machine. The only immediate benefit to replacing apache really is a bit less configuration complexity. The reality though, is that our apache configuration is already built in puppet so it's only a theoretical savings. If apache was unreliable or otherwise deficient then it'd be worth the effort.

The more performant python http servers are rather heavyweight dependencies to add and non-trivial amount of code to implement a decent configurable setup. I'm not against implementing something better but I think that a simple server will work for the most basic use-case and we can build something better if there is a demand for it in the future.

http://www.gunicorn.org/en/ seems interesting

https://github.com/gragas/simpletcp

One thing to investigate further is the use of git daemon.

I don't think this is really the direction we were planning to go in anymore.