Note: This RFC is a draft. It lives in my local namespace while I flesh out the details of my proposal. Comments are welcomed provided they are asking clarifying questions and not passing judgement at this early point in time.
Currently the JavaScript unit tests for all extension and skin repos are run together on every single commit that enters the integration pipeline. We basically use QUnit for integration testing. Yet Selenium is meant for integration testing and has been adopted in many repos. To make QUnit usable for integration testing we load a MediaWiki instance and launch a web browser which runs all the unit tests in Special:JavaScript/qunit
The proposal here is to stop this practice and move towards a setup where JavaScript unit tests are forced to be written in such a way that they:
1) are run in isolation without side effects from unit tests in other extensions.
2) they can be run from the command line quicker, without any setup steps (MediaWiki, LocalSettings edits)
To do this we would follow the path taken in MobileFrontend and Popups. MobileFrontend is currently running its unit tests in two modes - a local `npm run test:unit` and the current Special:JavaScript/qunit/plain method. We use webpack to build the tests for the latter environment and would live to standardise (T203137) and [[ https://gerrit.wikimedia.org/r/455052 | remove it. ]]
[] Code would be rewritten to use module.exports and requires. This requires either the adoption of webpack in complicated extensions or making use of the ResourceLoader improvements that will soon be available off the back of T133462
[] Tests for all MediaWiki extensions/core would gradually be rewritten using a new node library mw-node-qunit which can be run inside `npm test` which will be provided by the reading web team and provide the minimal mediawiki JS environment needed for testing. It will force the user to write tests in a sane way, by forcing them to stub calls to mw.msg and mw.config amongst other things that can impact the global state and have side effects.
[] The current test runner would be removed.
I argue that our current usage of QUnit in this way is problematic for several reasons.
1) Most of the tests run in QUnit are not integration tests. In fact, this is a nuisance to most developers. Currently a badly written unit test that touches global variables can break another unrelated extension. This happens frequently.
2) Running QUnit tests in this way is not catching real errors. Most errors are being uncovered these days using Selenium
3) The fact we have to launch a browser and MediaWiki instance makes these jobs slow, particularly when run locally on an install with multiple extensions needed to function. In MobileFrontend QUnit tests run in 10s, compared to 2 minutes on Jenkins.
4) The way we run QUnit is incompatible with most Node.js tooling. It is very difficult to add code coverage for JS when we run tests in this way.
5) The current infrastructure limits what we can test. To test a function it has to be made publicly accessible which usually involves exposing it unnecessarily on the `mw` global object, which adds unnecessary noise.
[1] If necessary, we can retain the use of QUnit for integration testing, but only for a small and limited set of tests.