Some parameters for the design of the format
* The test definitions are YAML files
* They follow the example of x-amples stanzas we use to define tests for RESTbase, e.g. https://github.com/wikimedia/citoid/blob/master/spec.yaml#L99. However, they are not bound to a path, as would be the case when integrating with Swagger.
* Each test file contains a "suite" of test cases, given as a list, plus some meta-informationm, and optionally a list of "local fixtures".
** local fixtures are exactly like tests** Each test suite can also specify a setup sequence, but theya list of requests that are executed before the tests in the file.
** for global fixtures,st cases. see belowThe entries in the sequence are request/response pairs.
* Each test case consists of a number of request/response pairs, plus some meta-information.
* Fixtures files contain a setup sequence, plus some meta-data.
Meta-info for test suites:
* flavor marker (REST, other)
* unique name (identifier)
* description (free text)
* need* required fixtures (list of identifiers) (see below for more infos)
* tags (list of identifiers) for filtering
Meta-info for test case:
* description (free text)
Meta-info for fixtures:
* needed* fixtures (list of identifiers) marker
* tags (list of identifiers) for filtering* unique name (identifier)
* description (free text)
* required fixtures (list of identifiers)
Variables/Placeholders:
* We'll need a placeholder syntax to inject values of variables defined by previous requestsfixtures or by the test runner.
** Variable names can be local to the test suite (randomized or from the setup sequence), or global (from a fixture). Global variables are qualified by prefixing their name with the name of the suitfixture that defined them.
* A test case or test suite can define a variable with randomized content, providing a variable name and a fixed prefix* Variables may be injected to the runner via the command line or a config file. The test runner will append a randomized suffix to the valueA typical use case for this would be the name and password of a "root" user on the target wiki that can be used to create fixtures that require elevated privileges.
* A test case or suite can assign a value todefine a variable based on the response of a request (howe with randomized content, exactly?providing a variable name and a fixed prefix. specifying a patch into a json structure?)The test runner will append a randomized suffix to the value.
* A test case or suite export a variable - that is, make it available for use by other test suites under it's qualified name. This is intended for use by fixtures.In a setup request (in a suite or fixture), E.g.variables can be defined based on the response of a request (how, a fixture that creates an admin user would export the randomized name,exactly? password, and edit token of that user as variablesspecifying a patch into a json structure?).
* Variables may also be injected to the runner via the command line or a config filedefined in fixtures are exported for use in test suites. They are exposed using their qualified name, which has the fixture's name as a prefix. A typical use case for this would be the name and password of a "root" user on the target wiki that can be used to create fixtures that require evlevated privilegesVariables defined in a test suite's setup sequence are local to that suite.
Additional thoughts:
* global fixtures are defined exactly like tests, and executed exactly like tests. They are "fixtures" only because they are used as such, that is, tests declare them as dependencies, causing them to be executed first.
** Fixtures may depend on other fixtures.
** We may want a convention for naming and placing fixtures.
* the request spec provides an URL suffix to be appended to the base URL provided as a parameter to the test runner. URL parameters may be part of this suffix, or may be specified separately as a JSON object (see below).
* for requests, we need to be able to specify method, URL parameters, and headers, as well as form fields when POSTing multipart/form-data data. And maybe even upload streams.
* for responses, we want to check the status, headers, and body. We'll want to match the body as a string or as a JSON structure. In the future, binary streams represented in HEX.
* we may want a way to choose between exact matches and regex mode