Scap3 should support rollback
Closed, ResolvedPublic

Description

From notes in {T109512#1611078}

Rollback

A rollback is an automatically detected failure, any failure that requires some human interaction would require a deploy to a specific revision. This is due to the ambiguity of expected behaviors in a rollback scenario.

The capistrano method for rollbacks and deploys seems sane and would make a rollback almost instantaneous, that is, having the actual deployed code be a symlink to a directory containing the currently deployed checkout, and simply creating a new directory and swapping symlinks as part of the deploy procedure. This allows a rollback (or a deploy) to happen as quickly as a symlink swap.

Rollback Steps:

  1. Use git rev-parse to determine the SHA1 of a revision inside of main.py's Deploy class—make targets ignorant of tags, HEADs, refs, etc
  2. Passing the SHA1 off to the targets which use git new-workdir to create a new working copy of the repo from a cache directory
  3. The new working copy is created in a directory named after the SHA1
  4. At the start of deploy-local on a given target, create a file called .in-progress at the same level as the cache directory containing the SHA1 to be deployed
  5. At the end of Deploy loop through all targets again to mv .in-progress .done
  6. If a failure is detected during a deploy, a rollback to the revision specified in .done (i.e., verifying the current symlink points to a directory named after the SHA1 inside .done) should represent the state the target repository was in pre-deploy
thcipriani added a project: Deployment-Systems.
thcipriani added subscribers: thcipriani, Aklapper, GWicke and 3 others.
dduvall triaged this task as "Normal" priority.Sep 4 2015, 9:55 PM
dduvall claimed this task.
dduvall set Security to None.
dduvall moved this task from Needs triage to Services MVP on the Scap board.

Change 236241 had a related patch set uploaded (by Dduvall):
Rename and simplify some git deploy functions

https://gerrit.wikimedia.org/r/236241

dduvall edited the task description. (Show Details)Sep 9 2015, 4:39 PM

Change 236241 merged by jenkins-bot:
Rename and simplify some git deploy functions

https://gerrit.wikimedia.org/r/236241

Change 238839 had a related patch set uploaded (by Dduvall):
Support atomic promotion and rollback

https://gerrit.wikimedia.org/r/238839

Change 238839 merged by jenkins-bot:
Support atomic promotion and rollback

https://gerrit.wikimedia.org/r/238839

Nice! This has proved to be useful in translatewiki.net too (see http://laxstrom.name/blag/2015/01/02/oregano-deployment-tool/ ).

dduvall closed this task as "Resolved".Sep 23 2015, 5:30 PM

Nice! This has proved to be useful in translatewiki.net too (see http://laxstrom.name/blag/2015/01/02/oregano-deployment-tool/ ).

What a great breakdown of link behavior among other things.

I wrote a simple tool, probably re-inventing the wheel for the hundredth time,

We feel the same way. :)

dduvall moved this task from Services MVP to Done on the Scap board.Sep 23 2015, 5:30 PM