Page MenuHomePhabricator

Execute checks after each stage of deployment
ClosedPublic

Authored by dduvall on Oct 7 2015, 12:01 AM.

Details

Maniphest Tasks
T109515: Scap3 should support post-deploy checks
Reviewers
thcipriani
mmodell
mobrovac
demon
Commits
rMWTS33524a3df44c: Execute checks after each stage of deployment
rMSCA33524a3df44c: Execute checks after each stage of deployment
Patch without arc
git checkout -b D11 && curl -L https://phabricator.wikimedia.org/D11?download=true | git apply
Summary

Checks are loaded in main.DeployLocal from YAML configuration under
the scap deploy subdirectory and run following execution of each stage.

Configuration takes the following form, where type is the type of
check to perform ('command' by default), command is a program (and its
arguments) that exits with non-zero status to signal check failure, and
stage is one of the main.DeployLocal stages for which the check will
execute.

checks:
  unique_check_name:
    type: command
    command: /path/to/command --with arguments
    stage: promote

New check types can be implemented by subclassing the checks.Check
class and registered for use in configuration via the
checks.checktype decorator. Validation can be added by implementing a
validate method.

@checks.checktype('icinga')
class IcingaCheck(checks.Check):
    def validate(self):
        # get/validate Icinga command

Checks are executed in parallel following the dispatch of a
main.DeployLocal stage. Any failed check should result in a rollback
prompt in main.Deploy.

The explicit check stage has been removed and the service port probe
has been moved to promote to ensure that the service has started
before executing checks.

Test Plan

Run tox.

Bug: T109515
Change-Id: Ia0118f766f51ddd05ae04fde90f40e5fdc04825f

Diff Detail

Repository
rMSCA Scap
Branch
T109515
Lint
Lint Errors
SeverityLocationCodeMessage
Errorscap/checks.py:59E1101No Member
Errorscap/checks.py:90E1101No Member
Errorscap/main.py:525E1101No Member
Errorscap/main.py:610E731PEP8 E731
Errorscap/main.py:666E1101No Member
Errortests/checks_test.py:11F0401Import Error
Errortests/log_test.py:8F0401Import Error
Warningscap/checks.py:34W0622Redefined Builtin
Warningscap/checks.py:93W0612Unused Variable
Warningscap/checks.py:162W0622Redefined Builtin
Warningtests/checks_test.py:77W0232No Init
Warningtests/checks_test.py:99W0212Protected Access
Advicescap/checks.py:27C0103Invalid Name
Advicescap/checks.py:30C0111Missing Docstring
Advicescap/checks.py:40C0111Missing Docstring
Advicescap/checks.py:47R0914Too Many Locals
Advicescap/checks.py:66C0111Missing Docstring
Advicescap/checks.py:76C0111Missing Docstring
Advicescap/checks.py:93C0103Invalid Name
Advicescap/checks.py:158C0103Invalid Name
Advicescap/checks.py:212C0103Invalid Name
Advicetests/checks_test.py:1C0111Missing Docstring
Advicetests/checks_test.py:14C0111Missing Docstring
Advicetests/checks_test.py:15C0111Missing Docstring
Advicetests/checks_test.py:41C0111Missing Docstring
Unit
No Test Coverage

Event Timeline

dduvall retitled this revision from to Execute checks after each stage of deployment.
dduvall updated this object.
dduvall edited the test plan for this revision. (Show Details)
dduvall added reviewers: mmodell, thcipriani, demon.

This is sooo much better than gerrit. I don't know how anyone could voluntarily use gerrit with differential available. Gerrit isn't even tolerable and using it almost amounts to torture.

mmodell edited edge metadata.

This looks really good @dduvall, nice work man!

This revision is now accepted and ready to land.Oct 7 2015, 12:50 PM

Nicely defined! This definitely lays a solid base for future work.

Only one real problem with config_deploy checks (see inline comments) (and also one nitpick, 'cause I am a picker of nits).

Eager to see this land :)

docs/scripts.rst
21

This is still the behavior of service_port in scap.cfg

scap/main.py
817

This will fail if there are config_deploy checks since self.rev_dir wouldn't yet exist.

dduvall edited edge metadata.

Fixed PEP8 warning about lambda assignment and removed pylint from .arclint.

Restored errant changes to docs/scripts.rst

dduvall added inline comments.
scap/main.py
817

Crap. I guess there's no way to run checks after anything but fetch and promote then, unless we change the implementation to pass YAML via the command line, but that seems gross and not worth the marginal benefit.

Deploy targets now retrieve checks.yaml over HTTP from the deploy host, allowing checks to be configured/performed for pre-check stages.

I also reimplemented tasks.check_port to use/accept a timeout in seconds and exposed that as service_timeout in the config.

thcipriani edited edge metadata.
This revision was automatically updated to reflect the committed changes.