Page MenuHomePhabricator

Running smoke tests during deployment
Open, LowPublic

Description

We should be able to run smoke tests (unit, API, Selenium...) during deployment. The question is how to do it.

#1 Smoke tests run from deployer's machine

  • Advantages:
    • No development needed.
    • Deployment documentation (example: SWAT deploys/Deployers) has an extra step where the deployer runs tests (example: npm run smoke) targeting a machine where the code is deployed (example: mwdebug1002)
  • Disadvantages:
    • Each deployer has to set up machine so every type of smoke tests would run.
    • A deployer might forget or decide not to run tests.

#2 Smoke tests run from scap

  • Advantages:
    • It happens automatically.
    • Deployers do not need any special setup on their machines.
  • Disadvantages:
    • Requires some development.

#3 Something else?