Requirements:

• The runner can be invoked from the command line
• Required input: the base URL of a MediaWiki instance
• Required input: one or more test description files.
  • See T219883: Draft file format for phester test definitions for the file format. The following features can be ignored for now: fixtures, variable, regular expression matches, tags. A setup section should however be supported.
• The test runner executes each test case against the given API and reports any failures to comply with the expected results
• human readable plain text output

Functional outline:

• Run the test suites in sequence, according to the order given on the command line
  • Execute the requests within each suite in sequence.

Additional notes and considerations:

• The test runner should be implemented in PHP. Rationale: It is intended to run in a development environment used to write PHP code. Also, we may want to pull this into the MediaWIki core project via composer at some point.
• Use the Guzzle library for making HTTP requests
• The test runner should not depend on MediaWiki core code.
• The test runner should not hard code any knowledge about the MediaWiki action API, and should be designed to be usable for testing other APIs, such as RESTbase.
• The test runner MUST ask for confirmation that it is ok for any information in the given target API to be damaged or lost (unless --force is specified)
• No cleanup (tear-down) is performed between tests.