Page MenuHomePhabricator

Draft file format for phester test definitions
Closed, ResolvedPublic1 Story Points

Description

Some parameters for the design of the format

  • The test definitions are YAML files
  • They follow the example of x-amples stanzas we use to define tests for RESTbase, e.g. https://github.com/wikimedia/citoid/blob/master/spec.yaml#L99. However, they are not bound to a path, as would be the case when integrating with Swagger.
  • Each test file contains a "suite" of test cases, given as a list, plus some meta-information.
    • Each test suite can also specify a setup sequence, a list of requests that are executed before the test cases. The entries in the sequence are request/response pairs.
  • Each test case consists of a number of request/response pairs, plus some meta-information.
  • Fixtures files contain a setup sequence, plus some meta-data.

Meta-info for test suites:

  • flavor marker (REST, RPC, GraphQL, SOAP, etc) -- we may not actually need this.
  • unique name (identifier)
  • description (free text)
  • required fixtures (list of identifiers)
  • tags (list of identifiers) for filtering

Meta-info for test case:

  • description (free text)

Meta-info for fixtures:

  • fixture marker
  • variable export list (list of identifiers).
  • unique name (identifier)
  • description (free text)
  • required fixtures (list of identifiers)

Variables/Placeholders:

  • We'll need a placeholder syntax to inject values of variables defined by fixtures or by the test runner.
    • Variable names can be local to the test suite (randomized or from the setup sequence), or global (from a fixture). Global variables are qualified by prefixing their name with the name of the fixture that defined them.
  • Variables may be injected to the runner via the command line or a config file. A typical use case for this would be the name and password of a "root" user on the target wiki that can be used to create fixtures that require elevated privileges.
  • A test suite can define a variable with randomized content, providing a variable name and a fixed prefix. The test runner will append a randomized suffix to the value.
  • Variables can be defined based on the response of a request (how, exactly? specifying a patch into a json structure?). This can be done within a test case to supply a CSRF token, or a global fixture to supply the ID of a user or page to test cases, etc.
  • Variables defined in fixtures are exported for use in test suites. They are exposed using their qualified name, which has the fixture's name as a prefix. Variables defined in a test suite's setup sequence are local to that suite.

Additional thoughts:

  • We may want a convention for naming and placing fixtures.
  • the request spec provides an URL suffix to be appended to the base URL provided as a parameter to the test runner. URL parameters may be part of this suffix, or may be specified separately as a JSON object (see below).
  • for requests, we need to be able to specify method, URL parameters, and headers, as well as form fields when POSTing multipart/form-data data. And maybe even upload streams.
  • for responses, we want to check the status, headers, and body. We'll want to match the body as a string or as a JSON structure. In the future, binary streams represented in HEX.
  • we may want a way to choose between exact matches and regex mode

Draft from etherpad: https://etherpad.wikimedia.org/p/api-tests-yaml

# Ticket for reference https://phabricator.wikimedia.org/T220037

# test suite example
suite: TestActionQuery
description: Testing action=query
type: RPC
use-fixtures: # List of required fixtures
  - "Project.xampleFixture"
  - "Project.xampleFixture2"
tags: # List of tags relevant to test suite
  - tag 1
  - tag 2

variables:
    - name: title1
      prefix: Whatever_ # the value will be randomized, something like Whatever_4ghuq34
    - name: title2
      prefix: Whatever_ # different randomized value

setup:
    - request: # I only put one request here, but it could be a sequence
        method: post
      query:
              action: edit # create a page. Note that tests must not modify the page in a way that interferes with other tests in the same suite. 
              title: $title1
              text: The content of the new page
    response:
      status 200 # if not given, 200 is always expected. We could also omit the response stanza entirely
    extract: # or "assign" or "set" or... soemthing?
      - name: title1id
        path: [ "edit", "pageid" ] # If we want to be fancy, we could use xpath or css selectors.

    #alternative approach:
    extract;
      edit:
         pageid:
            title1id # the value/leaft node is the name of the variable

    #or, directly in the resonse
    response:
      status: 200
      body: # if body is given as a structure, so Content-Type: application/json is implicitly required
        edit:
          pageid:
            !extract:title1id # the !extract: prefix specifies that we want to store this value as a variable. We could also use !set: or !as: or !remember:


tests:
# note: execution order of tests is undefined and should perhaps be randomized to flush out bad assumptions about that order.
    - description: Get information about Main Page
       interaction: # a single test can involve multiple requests/responses
         request:
            method: get
            path: /api.php #appended to the base URL passed to the runner. Should be paossibel to specify, but can be omitted/empty
            query:
              action: query
              prop: info
              titles: Main Page
              format: json
            headers:
              accept: application/json
          response:
            status: 200
            headers:
              content-type: !pcre/pattern:/application\/json/  #that's a clean way to specify regular expressions, but it may get annyoing
            body: !pcre/pattern:/.+/

          # we could extract variables here, for use in subsequent requests of the same test (not other tests in the same suite)

    - description: Get a list of files used in the Main Page
        interaction: # a single test can involve multiple requests/responses
        request:
            method: post
            form-data:
              action: query
              prop: images
              titles: $MediaWiki.PageWithImages.title # variable exported from the MediaWiki.PageWithImages fixture
              format: json
            headers:
              content-type: multipart/form-data
          response:
            status: 200
            headers:
              content-type: application/json # if we are explicit about regexes, we don't need them for plain string matches
            body: # we want to be able to specify a json structure here
              query:
                pages:
                  $MediaWiki.PageWithImages.id:  #we need variable expansion in keys as well!
          ....
....

# Naming schemes
# For fixtures
  MediaWiki.AdminUser
  MediaWiki.DeletedPage
  MediaWiki.PageWithThreeRevisions
  MediaWiki.ext.AbuseFilter.BlockUrl

#For variables exported from fixtures:
  MediaWiki.AdminUser.name
  MediaWiki.AdminUser.id
  MediaWiki.AdminUser.token
  MediaWiki.DeletedPage.title
  MediaWiki.PageWithThreeRevisions.revId1
  MediaWiki.PageWithThreeRevisions.revId2
  MediaWiki.ext.AbuseFilter.BlockUrl.url
  
......

# Example of page of fixture
#I think we just one want fixture per file.
fixture: MediaWiki.login_user
    description: Login a user
    use-fixtures:
      -"/login_token"
    interaction:
    -   request:
          method: post
          form-data:
            action: login
            lgname: user
            lgpassword: secret
            lgtoken: $MediaWiki.login_token.logintoken # variable from required fixtures
            format: json
          headers:
            accept: application/json
            # Content-Type: multipart/form-data is implied by the presence of the form-data field above
          # if nothing is specified about the response, status 200 is requried.
    -   request: # the fixture executes a sequence of requests
          ....

# this would be in a separate file
fixture: MediaWiki.login_token
    description: Get login token
    export:
      - logintoken # variable from response to save
    interaction:
       request:
          method: get
          query:
            action: query
            meta: tokens
            format: json
            type: login
          headers:
            accept: application/json
          extract:
              logintoken: [ "tokens", "logintoken" ] # that's not how login works, but good enough as an example :)

Event Timeline

daniel created this task.Apr 2 2019, 3:17 PM
daniel updated the task description. (Show Details)Apr 2 2019, 3:27 PM
Eevans added a subscriber: Eevans.EditedApr 2 2019, 3:39 PM

@daniel Are you aware of service-checker? It is invoked by Icinga and uses the x-amples stanza to monitor endpoint availability. We need this capability in Kask (session storage), but I've been reluctant to implement an OpenAPI/Swagger specification, just to have a data-structure to hang an x-amples off of. The alternative was to define an alternative structure, and then teach service-checker to use it.

Do you see any reason the format you've describe here could not serve both purposes?

daniel updated the task description. (Show Details)Apr 2 2019, 3:50 PM
daniel added a comment.Apr 2 2019, 3:52 PM

@Eevans I would hope that the structure I'm describing here could serve both purposes. The only issue I see is the environment. I want this to be implemented in PHP, so it's easy to run in a PHP dev environment, can be pulled in with composer, etc. But the mechanism should work for any kind of API, really.

daniel added a comment.EditedApr 2 2019, 3:55 PM

I discussed re-using service-checker with Marko, and it may be an option for CI, but could be a bit cumbersome locally. Our conclusion was that the functionality is trivial enough to make re-implementing the best choice.

But now, writing this spec, all the stuff about fixtures and variables is starting to look more and more complex. I had discarded StoryPlayer as overkill before. Perhaps we should re-consider, in case dealing with fixtures and variables and containers proves to be more work than we expected.

Eevans added a comment.Apr 2 2019, 4:19 PM

I discussed re-using service-checker with Marko, and it may be an option for CI, but could be a bit cumbersome locally. Our conclusion was that the functionality is trivial enough to make re-implementing the best choice.

Yeah, I could see where the execution models might be different enough to warrant different implementations.

But now, writing this spec, all the stuff about fixtures and variables is starting to look more and more complex. I had discarded StoryPlayer as overkill before. Perhaps we should re-consider, in case dealing with fixtures and variables and containers proves to be more work than we expected.

Could fixures be decoupled from the test specification? The former would seem to require a great deal more flexibility than the later; Would it be easier to implement them as code, rather than define a DSL to cover every possibility?

daniel added a comment.Apr 2 2019, 4:30 PM

Could fixures be decoupled from the test specification? The former would seem to require a great deal more flexibility than the later; Would it be easier to implement them as code, rather than define a DSL to cover every possibility?

I was thinking of using XML dumps to define fixtures, or simply SQL dumps. But XML dumps can't create user accounts, and SQL dumps won't for for extensions that need extra fixtures.

The DSL for fixtures really isn't different from the one for tests, if we assume that we can and will create all fixtures via the API. The only issue that complicates things is the need to have information flow from the responses in the fixture creation to the request in test cases. Most importantly, edit tokens, but also randomized names of users and page titles.

We could try to force fixed values, but that may not work for everything, and would be a major security risk if the tests are run against a publicly reachable wiki. This is the reason we still don't run extension Selenium tests against beta, I think. It would need to create admin accounts with well known names and passwords.

daniel updated the task description. (Show Details)Apr 2 2019, 4:32 PM
Eevans added a comment.Apr 2 2019, 9:29 PM

Could fixures be decoupled from the test specification? The former would seem to require a great deal more flexibility than the later; Would it be easier to implement them as code, rather than define a DSL to cover every possibility?

I was thinking of using XML dumps to define fixtures, or simply SQL dumps. But XML dumps can't create user accounts, and SQL dumps won't for for extensions that need extra fixtures.

My bad; For some reason I had the impression that fixtures went beyond application state, and included environment.

The DSL for fixtures really isn't different from the one for tests, if we assume that we can and will create all fixtures via the API. The only issue that complicates things is the need to have information flow from the responses in the fixture creation to the request in test cases. Most importantly, edit tokens, but also randomized names of users and page titles.

OK, that doesn't sound too terribly complex then. At a high-level, it sounds as if you'd parse the specification to create a directed graph of the test cases (including any fixtures), and then run through them while read/writing to a context that gets passed down the chain. The Devil is in the details though I suppose.

daniel added a comment.Apr 2 2019, 9:33 PM

OK, that doesn't sound too terribly complex then. At a high-level, it sounds as if you'd parse the specification to create a directed graph of the test cases (including any fixtures), and then run through them while read/writing to a context that gets passed down the chain. The Devil is in the details though I suppose.

This should be it. If it gets much more complex than that, I'd say we abort and re-think.

daniel updated the task description. (Show Details)Apr 4 2019, 11:13 AM
daniel updated the task description. (Show Details)Apr 8 2019, 3:11 PM
Pchelolo added a subscriber: Pchelolo.EditedApr 10 2019, 3:30 PM

Variables can be defined based on the response of a request (how, exactly? specifying a patch into a json structure?). This can be done within a test case to supply a CSRF token, or a global fixture to supply the ID of a user or page to test cases, etc.

We do have a very similar concept implemented in js for restbase handler templates. Basically, for each request sequence (a test case) we name each request, store the response as a property of an object and then reference them as a JSON path. This works pretty well.

WDoranWMF set the point value for this task to 1.Apr 15 2019, 3:10 PM
CCicalese_WMF triaged this task as Normal priority.Mon, Apr 29, 2:54 PM
daniel renamed this task from Specify file format for API test definitions to Draft file format for phester test definitions.
daniel removed a project: Code-Health.
daniel updated the task description. (Show Details)Mon, Apr 29, 7:53 PM
daniel closed this task as Resolved.

Drafting is done, finalization of the file format is now tracked as T222103: Finalize file format for phester test definitions.

Eevans updated the task description. (Show Details)Tue, Apr 30, 7:46 PM

ping @zeljkofilipin: you are already on the other relevant tickets and also on the spreadsheet, but I though you may want to have a look at the test file example here.