Some parameters for the design of the format
- The test definitions are YAML files
- They follow the example of x-amples stanzas we use to define tests for RESTbase, e.g. https://github.com/wikimedia/citoid/blob/master/spec.yaml#L99. However, they are not bound to a path, as would be the case when integrating with Swagger.
- Each test file contains a "suite" of test cases, given as a list, plus some meta-information.
- Each test suite can also specify a setup sequence, a list of requests that are executed before the test cases. The entries in the sequence are request/response pairs.
- Each test case consists of a number of request/response pairs, plus some meta-information.
- Fixtures files contain a setup sequence, plus some meta-data.
Meta-info for test suites:
- flavor marker (REST, RPC, GraphQL, SOAP, etc) -- we may not actually need this.
- unique name (identifier)
- description (free text)
- required fixtures (list of identifiers)
- tags (list of identifiers) for filtering
Meta-info for test case:
- description (free text)
Meta-info for fixtures:
- fixture marker
- variable export list (list of identifiers).
- unique name (identifier)
- description (free text)
- required fixtures (list of identifiers)
Variables/Placeholders:
- We'll need a placeholder syntax to inject values of variables defined by fixtures or by the test runner.
- Variable names can be local to the test suite (randomized or from the setup sequence), or global (from a fixture). Global variables are qualified by prefixing their name with the name of the fixture that defined them.
- Variables may be injected to the runner via the command line or a config file. A typical use case for this would be the name and password of a "root" user on the target wiki that can be used to create fixtures that require elevated privileges.
- A test suite can define a variable with randomized content, providing a variable name and a fixed prefix. The test runner will append a randomized suffix to the value.
- Variables can be defined based on the response of a request (how, exactly? specifying a patch into a json structure?). This can be done within a test case to supply a CSRF token, or a global fixture to supply the ID of a user or page to test cases, etc.
- Variables defined in fixtures are exported for use in test suites. They are exposed using their qualified name, which has the fixture's name as a prefix. Variables defined in a test suite's setup sequence are local to that suite.
Additional thoughts:
- We may want a convention for naming and placing fixtures.
- the request spec provides an URL suffix to be appended to the base URL provided as a parameter to the test runner. URL parameters may be part of this suffix, or may be specified separately as a JSON object (see below).
- for requests, we need to be able to specify method, URL parameters, and headers, as well as form fields when POSTing multipart/form-data data. And maybe even upload streams.
- for responses, we want to check the status, headers, and body. We'll want to match the body as a string or as a JSON structure. In the future, binary streams represented in HEX.
- we may want a way to choose between exact matches and regex mode
Draft from etherpad: https://etherpad.wikimedia.org/p/api-tests-yaml
# Ticket for reference https://phabricator.wikimedia.org/T220037 # test suite example suite: TestActionQuery description: Testing action=query type: RPC use-fixtures: # List of required fixtures - "Project.xampleFixture" - "Project.xampleFixture2" tags: # List of tags relevant to test suite - tag 1 - tag 2 variables: - name: title1 prefix: Whatever_ # the value will be randomized, something like Whatever_4ghuq34 - name: title2 prefix: Whatever_ # different randomized value setup: - request: # I only put one request here, but it could be a sequence method: post query: action: edit # create a page. Note that tests must not modify the page in a way that interferes with other tests in the same suite. title: $title1 text: The content of the new page response: status 200 # if not given, 200 is always expected. We could also omit the response stanza entirely extract: # or "assign" or "set" or... soemthing? - name: title1id path: [ "edit", "pageid" ] # If we want to be fancy, we could use xpath or css selectors. #alternative approach: extract; edit: pageid: title1id # the value/leaft node is the name of the variable #or, directly in the resonse response: status: 200 body: # if body is given as a structure, so Content-Type: application/json is implicitly required edit: pageid: !extract:title1id # the !extract: prefix specifies that we want to store this value as a variable. We could also use !set: or !as: or !remember: tests: # note: execution order of tests is undefined and should perhaps be randomized to flush out bad assumptions about that order. - description: Get information about Main Page interaction: # a single test can involve multiple requests/responses request: method: get path: /api.php #appended to the base URL passed to the runner. Should be paossibel to specify, but can be omitted/empty query: action: query prop: info titles: Main Page format: json headers: accept: application/json response: status: 200 headers: content-type: !pcre/pattern:/application\/json/ #that's a clean way to specify regular expressions, but it may get annyoing body: !pcre/pattern:/.+/ # we could extract variables here, for use in subsequent requests of the same test (not other tests in the same suite) - description: Get a list of files used in the Main Page interaction: # a single test can involve multiple requests/responses request: method: post form-data: action: query prop: images titles: $MediaWiki.PageWithImages.title # variable exported from the MediaWiki.PageWithImages fixture format: json headers: content-type: multipart/form-data response: status: 200 headers: content-type: application/json # if we are explicit about regexes, we don't need them for plain string matches body: # we want to be able to specify a json structure here query: pages: $MediaWiki.PageWithImages.id: #we need variable expansion in keys as well! .... .... # Naming schemes # For fixtures MediaWiki.AdminUser MediaWiki.DeletedPage MediaWiki.PageWithThreeRevisions MediaWiki.ext.AbuseFilter.BlockUrl #For variables exported from fixtures: MediaWiki.AdminUser.name MediaWiki.AdminUser.id MediaWiki.AdminUser.token MediaWiki.DeletedPage.title MediaWiki.PageWithThreeRevisions.revId1 MediaWiki.PageWithThreeRevisions.revId2 MediaWiki.ext.AbuseFilter.BlockUrl.url ...... # Example of page of fixture #I think we just one want fixture per file. fixture: MediaWiki.login_user description: Login a user use-fixtures: -"/login_token" interaction: - request: method: post form-data: action: login lgname: user lgpassword: secret lgtoken: $MediaWiki.login_token.logintoken # variable from required fixtures format: json headers: accept: application/json # Content-Type: multipart/form-data is implied by the presence of the form-data field above # if nothing is specified about the response, status 200 is requried. - request: # the fixture executes a sequence of requests .... # this would be in a separate file fixture: MediaWiki.login_token description: Get login token export: - logintoken # variable from response to save interaction: request: method: get query: action: query meta: tokens format: json type: login headers: accept: application/json extract: logintoken: [ "tokens", "logintoken" ] # that's not how login works, but good enough as an example :)