Now that we have a better understanding what phester should do and how much work it would be to create it, we should re-consider using existing tools instead.
An initial survey turned up the following options:
- strst (typescript, new/small project, yaml based, lacks some features)
- tavern (python, new/small project, yaml based, lacks some features)
- behat (php, established, feature rich & complex, focus on cucumber style logic tests)
- codeception (php, established, feature rich & complex, focus on fluent style logic tests)
- https://www.cypress.io/ based on node.js, tests are written in node.js; seems to want to avoid the kind of fixture creation described in the phester spec.
- https://robotframework.org/ written in python. Cucumber-ish tests
- https://www.soapui.org/open-source.html A drag-and drop UI approach to test specification
- https://github.com/apiaryio/dredd Written in node.js, tests are YAML files defining APIs. Mature
These should be discussed, and more should perhaps be found and considered.
Decision matrix (tentative): https://docs.google.com/spreadsheets/d/1G50XPisubSRttq4QhakSij8RDF5TBAxrJBwZ7xdBZG0/edit#gid=0
Requirements, that seem particular to (or especially important for) the Wikimedia use case:
- HTTP-centric paradigm, focusing on specifying the headers and body of requests, and running assertions against headers and body of the response.
- Support for running assertions against parts of a structured (JSON) response (JSON-to-JSON comparison, with the ability to use the more human friendly YAML syntax)
- filtering by tags (because we expect to have a large number of tests)
- parallel execution (because we expect to have a large number of tests)
- yaml based declarative tests and fixtures: tests should be language agnostic, it should be easy to write tests for people involved with different language ecosystems and code bases. This also avoids lock-in to a specific tool, since yaml is easy to parse and convert.
- generalized fixture creation, entirely API based, without the need to write "code" other than specifying requests in yaml.
- randomized fixtures, so we can create privileged users on potentially public tests systems.
- control over cookies and sessions
- ease of running on in dev environments without the need to install additional tools / infrastructure (this might by a reason to switch to python for implementation; node.js is also still in the race).
- discovery of tests defined by extensions.