- are run in isolation without side effects from unit tests in other extensions.
- they can be run from the command line quicker, without any setup steps (MediaWiki, LocalSettings edits)
I argue that our current usage of QUnit in this way is problematic for several reasons.
- Running QUnit tests in this way is not catching real integration errors. Most integration errors are being uncovered these days using Selenium
- The fact we have to launch a browser and MediaWiki instance makes these jobs slow, particularly when run locally on an install with multiple extensions needed to function. In MobileFrontend headless QUnit tests run in 10s, compared to 2 minutes on Jenkins (when all extensions are also run)
- The way we run QUnit is incompatible with most Node.js tooling. It is very difficult to add code coverage for JS when we run tests in this way.
- The current infrastructure limits what we can test. To test a function it has to be made publicly accessible which usually involves exposing it unnecessarily on the mw global object, which adds unnecessary noise.
- Code would be rewritten to use module.exports and requires. This requires either the adoption of webpack in complicated extensions or making use of the ResourceLoader improvements available off the back of T133462. I am focusing on the latter approach and have a proof of concept describing a backwards compatible migration. See: https://gerrit.wikimedia.org/r/#/c/mediawiki/core/+/487168/
- The majority of tests in MediaWiki extensions/core would gradually be rewritten using a new node library mw-node-qunit which can be run inside npm test which will be provided by the reading web team and provide the minimal mediawiki JS environment needed for testing. It will force the user to write tests in a sane way, by forcing them to stub calls to mw.msg and mw.config amongst other things that can impact the global state and have side effects.
- It should be possible for the tests to be run inside real browsers via Karma as well as in Node.js.
- The current test runner is kept for running true integration tests in QUnit where Selenium is not an option.
Q and A / talking points
Maybe. There might be value in using qunit for a small set of unit tests. I would argue that Selenium despite it's problems would be a better approach for ensuring compatibility between different extensions.
Wouldn't we lose protection against browser specific bugs?
I am not aware of any situations we have caught a regression by running our unit tests in different browsers e.g. Firefox/Chrome/Internet Explorer. While a nice idea, I don't see any evidence that we would catch any issues. It seems hypothetical but not practical. The majority of issues in the mobile website are caught via Selenium (I can collect data if we really need this). Selenium seems a much better way to catch these problems. Unit tests tend to prevent errors entering the codebase and protecting known errors from reappearing in the codebase. Unit tests for example did not help us prevent a recent regression in mobile T215536 while Selenium did.
 If necessary, we can retain the use of QUnit for integration testing, but only for a small and limited set of tests.