The pywikibot test suite is built to adjust to its working environment, skipping tests were dependencies are missing, and ignoring failing tests.

This has occasionally caused tests to be skipped/ignored when they should have run, such as due to Travis changes (T135701) or developers not updating decorators (T129368).

To solve this we need an advanced test runner (T115313) that can be used on the CI builds to ensure that a list of expected skips/expected failures for each job is maintained.

There are two acceptable ways to solve this:

• A plugin that exports the unit test results in a structured way (T135794), and then use a separate tool to check the unit test results for compliance with the maintained lists.
• A plugin that check the unit test results for compliance with the maintained lists at the end of the unit test run.

The former is a better design, separating the tasks, so that the analysis/validation tool does not need to be integrated into the test runner and could even use another language and hopefully even a pre-existing tool might have the desired functionality.

The latter has one advantage, in that it can also cater for allowing failures & errors, by returning a 'success' exit code when the only failures/errors are in a maintained list. However this could be the subject of a different feature.

What isnt acceptable is directly using special features only available in one test runner, and also undesirable is messing around inside the unit tests themselves to alter how they report depending on the CI environment (we already do this, and need to stop).