Now that we have a better understanding what phester should do and how much work it would be to create it, we should re-consider using existing tools instead.
An initial survey turned up the following options:
* [[https://github.com/eykrehbein/strest|strst]] (typescript, new/small project, yaml based, lacks some features)
* [[https://github.com/taverntesting/tavern|tavern]] (python, new/small project, yaml based, lacks some features)
* [[https://behat.org/|behat]] (php, established, feature rich & complex, focus on cucumber style logic tests)
* [[https://codeception.com/|codeception]] (php, established, feature rich & complex, focus on fluent style logic tests)
-------
* https://www.cypress.io/ based on node.js, tests are written in node.js; seems to want to avoid the kind of fixture creation described in the phester spec.
* https://robotframework.org/ written in python. Cucumber-ish tests
* https://www.soapui.org/open-source.html A drag-and drop UI approach to test specification
* https://github.com/apiaryio/dredd Written in node.js, tests are YAML files defining APIs. Mature
These should be discussed, and more should perhaps be found and considered.
Decision matrix (tentative): https://docs.google.com/spreadsheets/d/1G50XPisubSRttq4QhakSij8RDF5TBAxrJBwZ7xdBZG0/edit#gid=0
Requirements, that seem particular to (or especially important for) the Wikimedia use case:
* HTTP-centric paradigm, focusing on specifying the headers and body of requests, and running assertions against headers and body of the response.
* Support for running assertions against parts of a structured (JSON) response (JSON-to-JSON comparison, with the ability to use the more human friendly YAML syntax)
* filtering by tags (because we expect to have a large number of tests)
* parallel execution (because we expect to have a large number of tests)
* yaml based declarative tests and fixtures: tests should be language agnostic, it should be easy to write tests for people involved with different language ecosystems and code bases. This also avoids lock-in to a specific tool, since yaml is easy to parse and convert.
* generalized fixture creation, entirely API based, without the need to write "code" other than specifying requests in yaml.
* randomized fixtures, so we can create privileged users on potentially public tests systems.
* ease of running on in dev environments without the need to install additional tools / infrastructure (this might by a reason to switch to python for implementation; node.js is also still in the race).
* discovery of tests defined by extensions.