Page MenuHomePhabricator

Execute checks after each stage of deployment
ClosedPublic

Authored by dduvall on Oct 7 2015, 12:01 AM.

Details

Maniphest Tasks
T109515: Scap3 should support post-deploy checks
Reviewers
thcipriani
mmodell
mobrovac
demon
Commits
rMWTS33524a3df44c: Execute checks after each stage of deployment
rMSCA33524a3df44c: Execute checks after each stage of deployment
Patch without arc
git checkout -b D11 && curl -L https://phabricator.wikimedia.org/D11?download=true | git apply
Summary

Checks are loaded in main.DeployLocal from YAML configuration under
the scap deploy subdirectory and run following execution of each stage.

Configuration takes the following form, where type is the type of
check to perform ('command' by default), command is a program (and its
arguments) that exits with non-zero status to signal check failure, and
stage is one of the main.DeployLocal stages for which the check will
execute.

checks:
  unique_check_name:
    type: command
    command: /path/to/command --with arguments
    stage: promote

New check types can be implemented by subclassing the checks.Check
class and registered for use in configuration via the
checks.checktype decorator. Validation can be added by implementing a
validate method.

@checks.checktype('icinga')
class IcingaCheck(checks.Check):
    def validate(self):
        # get/validate Icinga command

Checks are executed in parallel following the dispatch of a
main.DeployLocal stage. Any failed check should result in a rollback
prompt in main.Deploy.

The explicit check stage has been removed and the service port probe
has been moved to promote to ensure that the service has started
before executing checks.

Test Plan

Run tox.

Bug: T109515
Change-Id: Ia0118f766f51ddd05ae04fde90f40e5fdc04825f

Diff Detail

Repository
rMSCA Scap
Branch
T109515
Lint
Lint Passed
Unit
No Test Coverage

Event Timeline

dduvall retitled this revision from to Execute checks after each stage of deployment.
dduvall updated this object.
dduvall edited the test plan for this revision. (Show Details)
dduvall added reviewers: mmodell, thcipriani, demon.

This is sooo much better than gerrit. I don't know how anyone could voluntarily use gerrit with differential available. Gerrit isn't even tolerable and using it almost amounts to torture.

mmodell edited edge metadata.

This looks really good @dduvall, nice work man!

This revision is now accepted and ready to land.Oct 7 2015, 12:50 PM

Nicely defined! This definitely lays a solid base for future work.

Only one real problem with config_deploy checks (see inline comments) (and also one nitpick, 'cause I am a picker of nits).

Eager to see this land :)

docs/scripts.rst
21

This is still the behavior of service_port in scap.cfg

scap/main.py
819

This will fail if there are config_deploy checks since self.rev_dir wouldn't yet exist.

dduvall edited edge metadata.

Fixed PEP8 warning about lambda assignment and removed pylint from .arclint.

Restored errant changes to docs/scripts.rst

dduvall added inline comments.
scap/main.py
819

Crap. I guess there's no way to run checks after anything but fetch and promote then, unless we change the implementation to pass YAML via the command line, but that seems gross and not worth the marginal benefit.

Deploy targets now retrieve checks.yaml over HTTP from the deploy host, allowing checks to be configured/performed for pre-check stages.

I also reimplemented tasks.check_port to use/accept a timeout in seconds and exposed that as service_timeout in the config.

thcipriani edited edge metadata.
This revision was automatically updated to reflect the committed changes.