Goal: Create end-to-end tests for MediaWiki API modules.
Motivation: Provide confidence for refactoring key parts of MediaWiki core
- A test runner loads test definitions and executes them against the API. We already have a python script that does this for x-amples tests for RESTbase, see https://github.com/wikimedia/operations-software-service-checker. It should be simple enough to extend, adopt, or port that.
- Test definitions for core are found by traversing a known directory in the MediaWiki repo. The location of test definitions for extensions can be specified in extension.json. We may also detect API tests found in some standard location in each extension directory, e.g. test/api or some such.
- The test definitions are yaml files that use the same structure as the x-amples stanzas we use to define tests for RESTbase, e.g. https://github.com/wikimedia/citoid/blob/master/spec.yaml#L99. However, they are defined per API module (action and query parameter), not per path as would be the case when integrating with Swagger, like in the example above.
- The test spec should probably specify the API flavor (“action” in our case), so it can later also be used with other flavors (e.g. “rest”).
- The test runner takes as a parameter the script URL of the wiki to test. It should be made abundantly clear that the target wiki is going to be polluted with test data.
- A docker image should be supplied that makes it easy to spin up an empty wiki for use as a test target. The mediawiki-docker-dev project could get a script that creates a dummy wiki, triggers the API tests, and tears down that test wiki once the tests are done.
- Fixtures (known system state) are created by running a number of API “tests” in sequence. We may want to make some boiler plate setup re-usable, to avoid copying the same tests over and over.
- Some tests (such as page deletion) require elevated privileges. The test runner should set up users with standardized names, and ensure they are logged in.
- This means there has to be a set of well known “tests” (read: API queries) that have to run before any other tests, to provide the global fixtures.
- Such global fixtures must not be modified. E.g. an API call that blocks a user must not block a user that is part of the global fixture. Instead, such a test must first create a user specifically for this test.
- No cleanup (tear-down) is performed between tests.
- Initially, test execution will be done manually in the development environment.
- We may want to set up a labs instance where we can run these tests against current master continuously. Alternatively, we could run them against the beta instances, along with selenium tests. Tests that need elevated privileges may however be problematic in that context.
- Ideally, we can run this in Jenkins. We do something similar for Selenium tests, so it should not be terribly hard to set up.
- Ideally, all relevant combinations of parameters of all API modules, including extensions
- Realistically, all parameters of all API modules in core that update the database. This goes beyond editing and includes watching pages, protecting pages, patrolling edits, blocking user, creating users, changing preferences, etc.
- Critical functionality that does not involve database updates also need baseline tests. This includes reading raw article content, fetching parsed article content, logging in, etc.
- We’ll want to test some key scenarios that go beyond per-module tests, like a blocked user trying to edit, or the autopatrolled flag.
- Critical API modules provided by extensions maintained and deployed by WMF should also be covered.