Page MenuHomePhabricator

Clean up QUnit test runner for MW (2020)
Open, MediumPublic

Description

It's high time to clean up a bit of technical debt. The below is a sequence of minor and incremental changes that together will hopefully result in the underlying model and intent being clearer to users, and also making QUnit tests easier to run and debug.

The below is about the built-in QUnit runner for MediaWiki, which includes support for integration tests and API tests. We currently use it to run "pure" unit tests as well. An RFC about the latter exists at T212521.

1. Offer a debug mode that uses packageFiles-style debug mode for all modules

The old debug mode tries to maintain 100% file references, and as such it has to load files raw which means the execution context changes from module scope to global scope. It also means that because there is no longer the ability to control execution time, each file has to be loaded in order one by one.

We can keep the 100% file reference parity option for those that want to keep this, but making the packageFiles-style debug mode available for everything is probably on the whole an improvement.

2. Enable debug mode by default in test context
  • Running in debug mode has the benefit of being easy to debug, and yet without requiring developers to switch contexts.
  • The bandwidth-saving measures of minification aren't needed in test context.
  • Disabling this will make the tests load even quicker in CI (currently loading and running all core QUnit tests takes about 4 seconds).
  • Disabling this will make the tests load faster in MediaWiki-Vagrant, which is (or was) known to have problems running the minifier in a timely fashion, sometimes timing out. I don't know if that's still the case today with PHP 7.2 and opcache, but one less thing to worry about either way.

This depends on step 1 first completing, as otherwise debug mode will cause behaviour differences compared to production (legacy global scope, and inefficient sequential file loading).

  • Enable debug mode by default in test context.
  • Remove the "Enable debug" checkbox from the test runner, it would be obsolete at this point.
3. Cut the last remaining LocalSettings influence

See also T89434: SpecialJavaScriptTest should not export page-specific mw.config
See also T212521#6028568.

Settings exposed in the test runner must never be influenced by LocalSettings.php. Any case of that happening is either a bug in the test case or a bug in core's test runner.

In the 2013 and 2015 refactors, the entire skin layer was gradaually phased out of the QUnit runner. But, one part remained for compatibility with a handful of tests that still needed it (against best practice).

The test runner is standalone and doesn't implement things like uselang or useskin (per the above). But because some extension tests relied (unintentionally) on indirectly asserting the output of an interface message, we kept the default of lang=en as ContentLanguage, and per T212521#6028568 that indeed means it can be influenced by LocalSettings.php

  1. ContentLanguage
    • Set ContentLanguage explicitly to en.
    • At this point, we wil have removed the last remaining LocalSettings.php influence that was accidentally left exposed in 2015 (namely wgContentLanguage) and thus makes local development consistent with WMF CI in terms of test behaviour.
    • Go further and disable localisation in test context entirely by switching from en to qqx. We already do this for many PHPUnit integration tests, and ResourceLoader internally has qqx as its default as wel..
      • Draft a commit that switches testrunner lang from en to qqx.
      • Fix whatever fails in CI.
  2. mw.config:
    • Draft a commit that removes the last remaining mw.config exports (effectively fixing T89434).
    • Fix whatever fails in CI by ensuring its values are mocked via the standard QUnit.newMwEnvironment (simple change), or where existing injection options exist, inject it instead.
    • Enjoy strict requirement that any mw.config values relied upon are declared by the test.
4. Add component dropdown

The "filter" and "module" parameters that QUnit offers by default (in the UI and via the query parameters) allow running of a specific test suite only instead of all tests, and to run all tests from a specific feature or extension (for example "mw.rcfilters" for all RCFilters test).

But, the string matching isn't very intuitive and becomes more difficult when the test suites don't have unique prefix. To make test selection easier, we can utilize the 'QUnitTestModule' attribute introduced in 2018 in extension.json. This removes the need for dealing with the QUnit.module() name and such, and instead allows you to select by the high-level components. For example "MediaWik core", "My Extension" or "My Skin".

5. Landing page

The current test page immediately starts loading and running the tests. This is fine if you decide what to run based on which exensions are enabled in LocalSettings.php (switch by commenting out etc.), or if you have used it before and already have the url with relevant filters applied.

But, when you run it for the first time and have numerous extenions enabled, this means you have to wait several seconds for most of these tests to run before the test runner becomes visible and interactive.

  • Make the test runner URL default to showinng the UI without running any tests.
  • When refreshing it should not require any kind of confirmation and still work as before. Same for CLI.

This means the default landing page will be where you select what to run and then run it.


After these smaller incremental changes, a possible next step (or done sooner / in parallel) could be to enable the Karma/Instanbul test coverage preset, but I'll leave that to be discussed at T212521 first.

Event Timeline

Krinkle created this task.Apr 12 2020, 10:28 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 12 2020, 10:28 PM
Krinkle moved this task from Inbox to QUnit on the MediaWiki-Core-Testing board.Apr 13 2020, 3:33 PM

Change 591988 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[mediawiki/core@master] resourceloader: Fix load.mock.php query parameter corruption in tests

https://gerrit.wikimedia.org/r/591988

Change 592039 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[mediawiki/core@master] qunit: Use "test.*" module names only for actual test suites

https://gerrit.wikimedia.org/r/592039

Change 592044 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[mediawiki/core@master] qunit: Load qunitjs directly instead of via load.php

https://gerrit.wikimedia.org/r/592044

Change 592046 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[mediawiki/core@master] qunit: Remove div#mw-javascripttest-summary from SpecialJavaScriptTest

https://gerrit.wikimedia.org/r/592046

Change 592050 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[mediawiki/core@master] qunit: Remove dead code from SpecialJavaScriptTest

https://gerrit.wikimedia.org/r/592050

Change 592083 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[mediawiki/core@master] mediawiki.jqueryMsg: Refactor test suite to not make any API requests

https://gerrit.wikimedia.org/r/592083

Change 592090 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[mediawiki/extensions/WikibaseLexeme@master] qunit: Remove unused 'wikibase.tests.qunit.testrunner' dependency

https://gerrit.wikimedia.org/r/592090

Change 592055 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[mediawiki/extensions/Wikibase@master] qunit: Remove QUnit.newWbEnvironment() abstraction layer

https://gerrit.wikimedia.org/r/592055

Change 592118 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[mediawiki/core@master] qunit: Disable mw.loader.store server-side instead of client-side

https://gerrit.wikimedia.org/r/592118

Change 592090 merged by jenkins-bot:
[mediawiki/extensions/WikibaseLexeme@master] qunit: Remove unused 'wikibase.tests.qunit.testrunner' dependency

https://gerrit.wikimedia.org/r/592090

Change 592055 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] qunit: Remove QUnit.newWbEnvironment() abstraction layer

https://gerrit.wikimedia.org/r/592055

Change 592039 merged by jenkins-bot:
[mediawiki/core@master] qunit: Use "test.*" module names only for actual test suites

https://gerrit.wikimedia.org/r/592039

Change 592044 merged by jenkins-bot:
[mediawiki/core@master] qunit: Load qunitjs directly instead of via load.php

https://gerrit.wikimedia.org/r/592044

Change 592046 merged by jenkins-bot:
[mediawiki/core@master] qunit: Remove div#mw-javascripttest-summary from SpecialJavaScriptTest

https://gerrit.wikimedia.org/r/592046

Change 592050 merged by jenkins-bot:
[mediawiki/core@master] qunit: Remove dead code from SpecialJavaScriptTest

https://gerrit.wikimedia.org/r/592050

Change 592118 merged by jenkins-bot:
[mediawiki/core@master] qunit: Disable mw.loader.store server-side instead of client-side

https://gerrit.wikimedia.org/r/592118

Change 592083 merged by jenkins-bot:
[mediawiki/core@master] mediawiki.jqueryMsg: Refactor test suite to not make any API requests

https://gerrit.wikimedia.org/r/592083

Change 591988 merged by jenkins-bot:
[mediawiki/core@master] resourceloader: Fix load.mock.php query parameter corruption in tests

https://gerrit.wikimedia.org/r/591988

Change 595331 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[mediawiki/core@master] mediawiki.base: Let exists() work even in lang=qqx

https://gerrit.wikimedia.org/r/595331

Change 592143 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[mediawiki/core@master] qunit: Load JS with lang=qqx/skin=fallback as context

https://gerrit.wikimedia.org/r/592143

Change 595331 merged by jenkins-bot:
[mediawiki/core@master] mediawiki.base: Let exists() work even in lang=qqx

https://gerrit.wikimedia.org/r/595331

Change 592143 merged by jenkins-bot:
[mediawiki/core@master] qunit: Load JS with lang=qqx/skin=fallback as context

https://gerrit.wikimedia.org/r/592143

Krinkle triaged this task as Medium priority.Mon, Jun 29, 1:35 AM