Scap3 should be able to deploy/rollback service config as part of deploy
Closed, ResolvedPublic2 Story Points

Description

Configuration deploy is currently handed by puppet. The process currently is:

  • puppet agent --disable
  • merge puppet changes to config
  • puppet agent --enable on 1 box
  • test changes on 1 machine
    • push any changes to puppet if needed
    • repeat until correct

This is currently burdensome since config changes and code changes often happen in concert, and this system presents onerous challenges in that case.

Success Criteria:

  • continue to utilize ops/puppet for private variable storage
  • decouple config changes from puppet runs
  • have config changes roll out with deploys
  • give deploy users the ability to diff config deploys (an ansible-like --check mechanism)
thcipriani updated the task description. (Show Details)
thcipriani raised the priority of this task from to Needs Triage.
thcipriani added a project: Deployments.
thcipriani added subscribers: thcipriani, GWicke, dduvall and 2 others.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 18 2015, 8:47 PM
thcipriani set Security to None.Aug 18 2015, 8:56 PM
thcipriani added a subscriber: mobrovac.
thcipriani triaged this task as Normal priority.

See my comment in T107532#1517098 for one idea that I proposed (using puppet apply for config changes) which I still believe to be fairly good compromise for this task.

During our pairing session Dan and I came up with a plan for implementing rollbacks:

Rollback

A rollback is an automatically detected failure, any failure that requires some human interaction would require a deploy to a specific revision. This is due to the ambiguity of expected behaviors in a rollback scenario.

The capistrano method for rollbacks and deploys seems sane and would make a rollback almost instantaneous, that is, having the actual deployed code be a symlink to a directory containing the currently deployed checkout, and simply creating a new directory and swapping symlinks as part of the deploy procedure. This allows a rollback (or a deploy) to happen as quickly as a symlink swap.

Rollback Steps:

  1. Use git rev-parse to determine the SHA1 of a revision inside of main.py's Deploy class—make targets ignorant of tags, HEADs, refs, etc
  2. Passing the SHA1 off to the targets which use git new-workdir to create a new working copy of the repo from a cache directory
  3. The new working copy is created in a directory named after the SHA1
  4. At the start of deploy-local on a given target, create a file called .in-progress at the same level as the cache directory containing the SHA1 to be deployed
  5. At the end of Deploy loop through all targets again to mv .in-progress .done
  6. If a failure is detected during a deploy, a rollback to the revision specified in .done (i.e., verifying the current symlink points to a directory named after the SHA1 inside .done) should represent the state the target repository was in pre-deploy

Change 240292 had a related patch set uploaded (by Thcipriani):
Add config deployment

https://gerrit.wikimedia.org/r/240292

Change 240292 merged by jenkins-bot:
Add config deployment

https://gerrit.wikimedia.org/r/240292

mmodell edited a custom field.Oct 12 2015, 4:15 PM
dduvall closed this task as Resolved.Oct 22 2015, 10:40 PM
dduvall moved this task from Services MVP to Done on the Scap board.