Page MenuHomePhabricator

Bring co-master / fanout capabilities to scap3 deployments
Closed, DeclinedPublic


Right now the co-master support and fanouts (dsh_masters, dsh_proxies) only work on scap but not deploy. This needs fixing obviously before we move MW over to deploy.

Event Timeline

demon raised the priority of this task from to Needs Triage.
demon updated the task description. (Show Details)
demon added a project: Scap.
demon added a subscriber: demon.
dduvall triaged this task as High priority.Feb 12 2016, 7:51 PM
dduvall set Security to None.
dduvall moved this task from Needs triage to MediaWiki MVP on the Scap board.
thcipriani lowered the priority of this task from High to Medium.Mar 4 2016, 6:48 PM

The current thinking is to integrate something very similar to into the scap repository.

Fanout nodes would then spin up gpack to serve traffic to further groups of nodes. Since this process is run by an unprivileged user, the wsgi server would have to spin up on a high port number that is accessible to the remaining nodes in the deploy group.

Would be nice to get some ops feedback on this idea since it would require a range of ports (to allow parallel fanout deploys of multiple repos) to be opened to internal machines.

Talked a bit with @fgiunchedi about this Monday, he asked us to open up the discussion on phab.

Why shouldn't we just use apache for this? It's easy to set up a specific virtual host on a high port for every repository which is configured via a proxy to serve static git files from a specific directory only to selected nodes.

What would the advantage be in using gpack instead?

following up here too, yesterday at the meeting I've also floated the idea of using a generic http service like swift to serve the git repos. As per T64835: Setup a Swift cluster on beta-cluster to match production there is a swift cluster in beta/deployment-prep now that can be used for experiments.

thcipriani renamed this task from Bring co-master / fanout capabilities to `deploy` and friends to Bring co-master / fanout capabilities to scap3 deployments.Oct 12 2016, 4:15 PM

I've just discovered uftp which is a very interesting multicast file transfer tool. It seems just about ideal for distributing code to a bunch of servers in parallel while avoiding overloading the deployment server.