Some parameters for the design of the format
* The test definitions are YAML files
* They follow the example of x-amples stanzas we use to define tests for RESTbase, e.g. https://github.com/wikimedia/citoid/blob/master/spec.yaml#L99. However, they are not bound to a path, as would be the case when integrating with Swagger.
* Each test file contains a "suite" of test cases, given as a list, plus some meta-information.
** Each test suite can also specify a setup sequence, a list of requests that are executed before the test cases. The entries in the sequence are request/response pairs.
* Each test case consists of a number of request/response pairs, plus some meta-information.
* Fixtures files contain a setup sequence, plus some meta-data.
Meta-info for test suites:
* flavor marker (REST, RPC, GraphQL, SOAP, etc) -- we may not actually need this.
* unique name (identifier)
* description (free text)
* required fixtures (list of identifiers)
* tags (list of identifiers) for filtering
Meta-info for test case:
* description (free text)
Meta-info for fixtures:
* fixture marker
* variable export list (list of identifiers).
* unique name (identifier)
* description (free text)
* required fixtures (list of identifiers)
Variables/Placeholders:
* We'll need a placeholder syntax to inject values of variables defined by fixtures or by the test runner.
** Variable names can be local to the test suite (randomized or from the setup sequence), or global (from a fixture). Global variables are qualified by prefixing their name with the name of the fixture that defined them.
* Variables may be injected to the runner via the command line or a config file. A typical use case for this would be the name and password of a "root" user on the target wiki that can be used to create fixtures that require elevated privileges.
* A test suite can define a variable with randomized content, providing a variable name and a fixed prefix. The test runner will append a randomized suffix to the value.
* Variables can be defined based on the response of a request (how, exactly? specifying a patch into a json structure?). This can be done within a test case to supply a CSRF token, or a global fixture to supply the ID of a user or page to test cases, etc.
* Variables defined in fixtures are exported for use in test suites. They are exposed using their qualified name, which has the fixture's name as a prefix. Variables defined in a test suite's setup sequence are local to that suite.
Additional thoughts:
* We may want a convention for naming and placing fixtures.
* the request spec provides an URL suffix to be appended to the base URL provided as a parameter to the test runner. URL parameters may be part of this suffix, or may be specified separately as a JSON object (see below).
* for requests, we need to be able to specify method, URL parameters, and headers, as well as form fields when POSTing multipart/form-data data. And maybe even upload streams.
* for responses, we want to check the status, headers, and body. We'll want to match the body as a string or as a JSON structure. In the future, binary streams represented in HEX.
* we may want a way to choose between exact matches and regex mode
Draft from etherpad: https://etherpad.wikimedia.org/p/api-tests-yaml
```
# Ticket for reference https://phabricator.wikimedia.org/T220037
# test suite example
suite: TestActionQuery
description: Testing action=query
type: RPC
use-fixtures: # List of required fixtures
- "Project.xampleFixture"
- "Project.xampleFixture2"
tags: # List of tags relevant to test suite
- tag 1
- tag 2
variables:
- name: title1
prefix: Whatever_ # the value will be randomized, something like Whatever_4ghuq34
- name: title2
prefix: Whatever_ # different randomized value
setup:
- request: # I only put one request here, but it could be a sequence
method: post
query:
action: edit # create a page. Note that tests must not modify the page in a way that interferes with other tests in the same suite.
title: $title1
text: The content of the new page
response:
status 200 # if not given, 200 is always expected. We could also omit the response stanza entirely
extract: # or "assign" or "set" or... soemthing?
- name: title1id
path: [ "edit", "pageid" ] # If we want to be fancy, we could use xpath or css selectors.
#alternative approach:
extract;
edit:
pageid:
title1id # the value/leaft node is the name of the variable
#or, directly in the resonse
response:
status: 200
body: # if body is given as a structure, so Content-Type: application/json is implicitly required
edit:
pageid:
!extract:title1id # the !extract: prefix specifies that we want to store this value as a variable. We could also use !set: or !as: or !remember:
tests:
# note: execution order of tests is undefined and should perhaps be randomized to flush out bad assumptions about that order.
- description: Get information about Main Page
interaction: # a single test can involve multiple requests/responses
request:
method: get
path: /api.php #appended to the base URL passed to the runner. Should be paossibel to specify, but can be omitted/empty
query:
action: query
prop: info
titles: Main Page
format: json
headers:
accept: application/json
response:
status: 200
headers:
content-type: !pcre/pattern:/application\/json/ #that's a clean way to specify regular expressions, but it may get annyoing
body: !pcre/pattern:/.+/
# we could extract variables here, for use in subsequent requests of the same test (not other tests in the same suite)
- description: Get a list of files used in the Main Page
interaction: # a single test can involve multiple requests/responses
request:
method: post
form-data:
action: query
prop: images
titles: $MediaWiki.PageWithImages.title # variable exported from the MediaWiki.PageWithImages fixture
format: json
headers:
content-type: multipart/form-data
response:
status: 200
headers:
content-type: application/json # if we are explicit about regexes, we don't need them for plain string matches
body: # we want to be able to specify a json structure here
query:
pages:
$MediaWiki.PageWithImages.id: #we need variable expansion in keys as well!
....
....
# Naming schemes
# For fixtures
MediaWiki.AdminUser
MediaWiki.DeletedPage
MediaWiki.PageWithThreeRevisions
MediaWiki.ext.AbuseFilter.BlockUrl
#For variables exported from fixtures:
MediaWiki.AdminUser.name
MediaWiki.AdminUser.id
MediaWiki.AdminUser.token
MediaWiki.DeletedPage.title
MediaWiki.PageWithThreeRevisions.revId1
MediaWiki.PageWithThreeRevisions.revId2
MediaWiki.ext.AbuseFilter.BlockUrl.url
......
# Example of page of fixture
#I think we just one want fixture per file.
fixture: MediaWiki.login_user
description: Login a user
use-fixtures:
-"/login_token"
interaction:
- request:
method: post
form-data:
action: login
lgname: user
lgpassword: secret
lgtoken: $MediaWiki.login_token.logintoken # variable from required fixtures
format: json
headers:
accept: application/json
# Content-Type: multipart/form-data is implied by the presence of the form-data field above
# if nothing is specified about the response, status 200 is requried.
- request: # the fixture executes a sequence of requests
....
# this would be in a separate file
fixture: MediaWiki.login_token
description: Get login token
export:
- logintoken # variable from response to save
interaction:
request:
method: get
query:
action: query
meta: tokens
format: json
type: login
headers:
accept: application/json
extract:
logintoken: [ "tokens", "logintoken" ] # that's not how login works, but good enough as an example :)
```