Page MenuHomePhabricator

Improve phabricator's deployment process
Open, MediumPublic

Description

In order for Phabricator to be sanely deployed using scap, a few things need to change, either in rMSCA Scap or in operations/puppet/modules/phabricator.

Problems with the current situation

After a scap deployment, we have to run puppet to get the deployed code in shape because puppet enforces a lot of permissions, creates config files, etc. So a newly deployed tag will be missing a bunch of things.

A second problem is that phabricator upgrades involve database migrations which must be coordinated with the code upgrade - once the migration is applied the old code is no longer valid and the migrations cannot be easily rolled back.

So the procedure looks like this:

  1. Stop phd
  2. Deploy new code version
  3. run puppet manually
  4. apply db migrations
  5. immediately reload apache to pick up the new code
  6. start phd

Proposed solution 1: replace manual steps with scap checks

I've attempted to break this up into 3 scripts which can be called from scap "checks"

Roughly divided as follows:

  • Promote
    1. Stop phd
    2. Deploy new code version
  • Finalize
    1. run puppet manually
    2. apply db migrations
    3. immediately reload apache to pick up the new code
    4. start phd
  • Rollback
    1. reload apache
    2. start phd
    3. hope that there weren't partially applied migrations which cause the old phabricator revision to be broken.

This far from ideal, it's essentially how things have been since day one with phabricator only a bit more automated.

Proposed solution 2: change scap

This one is covered in T172486: scap should provide a way to skip symlink-swapping in promote

Proposed solution 3: change rOPUP Wikimedia Puppet/modules/phabricator

After discussing this with @thcipriani, we decided that the best approach would be to change puppet instead of adding a new feature to scap to keep untracked files. I still think that would be nice to have in scap, however, it's not trivial given that scap does a fresh clone of the repo for every deployment.

I've submitted a first draft of solution #3 here

Event Timeline

Change 370622 had a related patch set uploaded (by 20after4; owner: 20after4):
[operations/puppet@production] PHAB: deployment scripts to be called by scap

https://gerrit.wikimedia.org/r/370622

mmodell triaged this task as Medium priority.Aug 9 2017, 5:21 AM
greg added a subscriber: greg.

(moving to Release-Engineering-Team (Next), feel free to move to -kanban if this is something on the shorter term)

Talked about this with @thcipriani and it looks like we can do it without touching scap - but it involves a bunch of puppet changes to move config files to a separate location and replacing them with symlinks in the repo.

This is the right way to solve it really so I'll have to bite the bullet and get the puppet changes through ops review.

mmodell updated the task description. (Show Details)

Change 374054 had a related patch set uploaded (by 20after4; owner: 20after4):
[operations/puppet@production] Move phabricator conf files outside of source tree

https://gerrit.wikimedia.org/r/374054

Change 374054 had a related patch set uploaded (by 20after4; owner: 20after4):
[operations/puppet@production] Move phabricator conf files outside of source tree

https://gerrit.wikimedia.org/r/374054

immediately reload apache to pick up the new code

IIRC the DB migration scripts says that apache should stopped before doing upgrades?

immediately reload apache to pick up the new code

IIRC the DB migration scripts says that apache should stopped before doing upgrades?

Yes, it's probably not terribly critical but it's a good idea.

In case it's useful: There's a read-only mode too.

Change 370622 merged by Dzahn:
[operations/puppet@production] PHAB: deployment scripts to be called by scap

https://gerrit.wikimedia.org/r/370622

Change 374054 merged by Dzahn:
[operations/puppet@production] Move phabricator conf files outside of source tree

https://gerrit.wikimedia.org/r/374054