Page MenuHomePhabricator

Local config deploys should use the target's current version
Closed, ResolvedPublic

Description

When executing scap deploy-local config-deploy on a target node, Scap3 asks the deployment server for the revision. However, it should use the current revision of the target instead. This is problematic in situations where only a portion of the nodes have a specific revision checked out, since the config version will match that of the last-deployed revision, not the one currently being used by the target.

As an example, consider a situation where out of 10 nodes, 5 have revA checked out and 5 have revB. If revB was the last to be deployed, then executing scap deploy-local config-deploy on the nodes that have revA checked out will result in them running revA, but paired with the config coming from revB.

Moreover, such an approach creates problems for normal deploys that happen in such scenarios. In the above example, a subsequent deployment of revB fails because Scap3 creates the revB cache directory, which makes git complain during the fetch phase:

10:00:15 ['/usr/bin/scap', 'deploy-local', '-v', '--repo', 'mathoid/deploy', '--force', '-g', 'default', 'fetch'] on scb2002.codfw.wmnet returned [70]: http://tin.eqiad.wmnet/mathoid/deploy/.git
From http://tin.eqiad.wmnet/mathoid/deploy/
 * [new branch]      master     -> origin/master
 * [new tag]         scap/sync/2016-09-12/0004 -> scap/sync/2016-09-12/0004
 * [new tag]         scap/sync/2016-09-12/0005 -> scap/sync/2016-09-12/0005
 * [new tag]         scap/sync/2016-09-12/0006 -> scap/sync/2016-09-12/0006
fatal: destination path '/srv/deployment/mathoid/deploy-cache/revs/9937e0fec4a26641b74be6709f17f21e236b193f' already exists and is not an empty directory.

Revisions and Commits

rMSCA Scap
Restricted Differential Revision

Event Timeline

thcipriani triaged this task as Medium priority.Sep 15 2016, 4:27 PM
thcipriani moved this task from Needs triage to Services improvements on the Scap board.

One plan of attack here is to make .git/DEPLOY_HEAD an actual symbolic ref and keep a history of deployments as git objects...somewhere. Either on a local branch or as a custom ref I suppose. This way we have a history of deployed configs that we can fetch on remotes and refer to by sha1. Also gives us lots of stuff for free.

See a log of deployments by doing:

git log DEPLOY_HEAD

And see the current deploy with:

git show DEPLOY_HEAD
thcipriani raised the priority of this task from Medium to High.Sep 19 2016, 5:38 PM

Per today's deployment-tooling meeting, the easiest path forward here might be to cache DEPLOY_HEAD on the deployment targets for config_deploy.

The plan mentioned above would take longer to implement, but would be nice to have. That plan is captured in T146062: DEPLOY_HEAD should be a symbolic ref

thcipriani added a revision: Restricted Differential Revision.Sep 21 2016, 10:20 PM