It's high time to clean up a bit of technical debt. The below is a sequence of minor and incremental changes that together will hopefully result in the underlying model and intent being clearer to users, and also making QUnit tests easier to run and debug.
The below is about the built-in QUnit runner for MediaWiki, which includes support for integration tests and API tests. We currently use it to run "pure" unit tests as well. An RFC about the latter exists at T212521.
1. Offer a debug mode that uses packageFiles-style debug mode for all modules
The old debug mode tries to maintain 100% file references, and as such it has to load files raw which means the execution context changes from module scope to global scope. It also means that because there is no longer the ability to control execution time, each file has to be loaded in order one by one.
We can keep the 100% file reference parity option for those that want to keep this, but making the packageFiles-style debug mode available for everything is probably on the whole an improvement.
2. Enable debug mode by default in test context
- Running in debug mode has the benefit of being easy to debug, and yet without requiring developers to switch contexts.
- The bandwidth-saving measures of minification aren't needed in test context.
- Disabling this will make the tests load even quicker in CI (currently loading and running all core QUnit tests takes about 4 seconds).
- Disabling this will make the tests load faster in MediaWiki-Vagrant, which is (or was) known to have problems running the minifier in a timely fashion, sometimes timing out. I don't know if that's still the case today with PHP 7.2 and opcache, but one less thing to worry about either way.
This depends on step 1 first completing, as otherwise debug mode will cause behaviour differences compared to production (legacy global scope, and inefficient sequential file loading).
- Enable debug mode by default in test context.
- Remove the "Enable debug" checkbox from the test runner, it would be obsolete at this point.
3. Cut the last remaining LocalSettings influence
See also T89434: SpecialJavaScriptTest should not export page-specific mw.config
See also T212521#6028568.
Settings exposed in the test runner must never be influenced by LocalSettings.php. Any case of that happening is either a bug in the test case or a bug in core's test runner.
In the 2013 and 2015 refactors, the entire skin layer was gradaually phased out of the QUnit runner. But, one part remained for compatibility with a handful of tests that still needed it (against best practice).
The test runner is standalone and doesn't implement things like uselang or useskin (per the above). But because some extension tests relied (unintentionally) on indirectly asserting the output of an interface message, we kept the default of lang=en as ContentLanguage, and per T212521#6028568 that indeed means it can be influenced by LocalSettings.php
- ContentLanguage
- Set ContentLanguage explicitly to en instead of inheriting developer's local settings.
- At this point, we wil have removed the last remaining LocalSettings.php influence that was accidentally left exposed in 2015 (namely wgContentLanguage) and thus makes local development consistent with WMF CI in terms of test behaviour.
- Disable localisation in test context entirely by switching from en to qqx and fix any would-be CI failures.
- We already do this for many PHPUnit integration tests, and ResourceLoader internally has qqx as its default as wel..
- mw.config:
- Draft a commit that removes the last remaining mw.config exports (effectively fixing T89434).
- Fix whatever fails in CI by ensuring its values are mocked via the standard QUnit.newMwEnvironment (simple change), or where existing injection options exist, inject it instead.
- Enjoy strict requirement that any mw.config values relied upon are declared by the test.
4. Add component dropdown
The "filter" and "module" parameters that QUnit offers by default (in the UI and via the query parameters) allow running of a specific test suite only instead of all tests, and to run all tests from a specific feature or extension (for example "mw.rcfilters" for all RCFilters test).
But, the string matching isn't very intuitive and becomes more difficult when the test suites don't have unique prefix. To make test selection easier, we can utilize the 'QUnitTestModule' attribute introduced in 2018 in extension.json. This removes the need for dealing with the QUnit.module() name and such, and instead allows you to select by the high-level components. For example "MediaWik core", "My Extension" or "My Skin".
- Add "component" parameter for loading and running tests based on the ExtensionRegistry.
- Add it to the UI.
- Add it to the CLI (e.g. npm run qunit -- EventLogging)
- Document it at https://www.mediawiki.org/wiki/Manual:JavaScript_unit_testing
- Also document filter/module parameters there (link to https://api.qunitjs.com/config/?)
5. Landing page
The current test page immediately starts loading and running the tests. This is fine if you decide what to run based on which exensions are enabled in LocalSettings.php (switch by commenting out etc.), or if you have used it before and already have the url with relevant filters applied.
But, when you run it for the first time and have numerous extenions enabled, this means you have to wait several seconds for most of these tests to run before the test runner becomes visible and interactive.
- Make the test runner URL default to showinng the UI without running any tests.
- When refreshing it should not require any kind of confirmation and still work as before. Same for CLI.
This means the default landing page will be where you select what to run and then run it.
After these smaller incremental changes, a possible next step (or done sooner / in parallel) could be to enable the Karma/Instanbul test coverage preset, but I'll leave that to be discussed at T212521 first.