Page MenuHomePhabricator

Clean up QUnit test runner for MW (2020)
Open, MediumPublic

Description

It's high time to clean up a bit of technical debt. The below is a sequence of minor and incremental changes that together will hopefully result in the underlying model and intent being clearer to users, and also making QUnit tests easier to run and debug.

The below is about the built-in QUnit runner for MediaWiki, which includes support for integration tests and API tests. We currently use it to run "pure" unit tests as well. An RFC about the latter exists at T212521.

1. Offer a debug mode that uses packageFiles-style debug mode for all modules

The old debug mode tries to maintain 100% file references, and as such it has to load files raw which means the execution context changes from module scope to global scope. It also means that because there is no longer the ability to control execution time, each file has to be loaded in order one by one.

We can keep the 100% file reference parity option for those that want to keep this, but making the packageFiles-style debug mode available for everything is probably on the whole an improvement.

2. Enable debug mode by default in test context
  • Running in debug mode has the benefit of being easy to debug, and yet without requiring developers to switch contexts.
  • The bandwidth-saving measures of minification aren't needed in test context.
  • Disabling this will make the tests load even quicker in CI (currently loading and running all core QUnit tests takes about 4 seconds).
  • Disabling this will make the tests load faster in MediaWiki-Vagrant, which is (or was) known to have problems running the minifier in a timely fashion, sometimes timing out. I don't know if that's still the case today with PHP 7.2 and opcache, but one less thing to worry about either way.

This depends on step 1 first completing, as otherwise debug mode will cause behaviour differences compared to production (legacy global scope, and inefficient sequential file loading).

  • Enable debug mode by default in test context.
  • Remove the "Enable debug" checkbox from the test runner, it would be obsolete at this point.
3. Cut the last remaining LocalSettings influence

See also T89434: SpecialJavaScriptTest should not export page-specific mw.config
See also T212521#6028568.

Settings exposed in the test runner must never be influenced by LocalSettings.php. Any case of that happening is either a bug in the test case or a bug in core's test runner.

In the 2013 and 2015 refactors, the entire skin layer was gradaually phased out of the QUnit runner. But, one part remained for compatibility with a handful of tests that still needed it (against best practice).

The test runner is standalone and doesn't implement things like uselang or useskin (per the above). But because some extension tests relied (unintentionally) on indirectly asserting the output of an interface message, we kept the default of lang=en as ContentLanguage, and per T212521#6028568 that indeed means it can be influenced by LocalSettings.php

  1. ContentLanguage
    • Set ContentLanguage explicitly to en.
    • At this point, we wil have removed the last remaining LocalSettings.php influence that was accidentally left exposed in 2015 (namely wgContentLanguage) and thus makes local development consistent with WMF CI in terms of test behaviour.
    • Go further and disable localisation in test context entirely by switching from en to qqx. We already do this for many PHPUnit integration tests, and ResourceLoader internally has qqx as its default as wel..
      • Draft a commit that switches testrunner lang from en to qqx.
      • Fix whatever fails in CI.
  2. mw.config:
    • Draft a commit that removes the last remaining mw.config exports (effectively fixing T89434).
    • Fix whatever fails in CI by ensuring its values are mocked via the standard QUnit.newMwEnvironment (simple change), or where existing injection options exist, inject it instead.
    • Enjoy strict requirement that any mw.config values relied upon are declared by the test.
4. Add component dropdown

The "filter" and "module" parameters that QUnit offers by default (in the UI and via the query parameters) allow running of a specific test suite only instead of all tests, and to run all tests from a specific feature or extension (for example "mw.rcfilters" for all RCFilters test).

But, the string matching isn't very intuitive and becomes more difficult when the test suites don't have unique prefix. To make test selection easier, we can utilize the 'QUnitTestModule' attribute introduced in 2018 in extension.json. This removes the need for dealing with the QUnit.module() name and such, and instead allows you to select by the high-level components. For example "MediaWik core", "My Extension" or "My Skin".

5. Landing page

The current test page immediately starts loading and running the tests. This is fine if you decide what to run based on which exensions are enabled in LocalSettings.php (switch by commenting out etc.), or if you have used it before and already have the url with relevant filters applied.

But, when you run it for the first time and have numerous extenions enabled, this means you have to wait several seconds for most of these tests to run before the test runner becomes visible and interactive.

  • Make the test runner URL default to showinng the UI without running any tests.
  • When refreshing it should not require any kind of confirmation and still work as before. Same for CLI.

This means the default landing page will be where you select what to run and then run it.


After these smaller incremental changes, a possible next step (or done sooner / in parallel) could be to enable the Karma/Instanbul test coverage preset, but I'll leave that to be discussed at T212521 first.

Details

ProjectBranchLines +/-Subject
mediawiki/coremaster+15 -66
mediawiki/coremaster+5 -48
mediawiki/extensions/GrowthExperimentsmaster+3 -3
mediawiki/extensions/CentralNoticewmf_deploy+6 -6
mediawiki/coremaster+26 -26
mediawiki/extensions/WikibaseLexememaster+2 -2
mediawiki/skins/MinervaNeuemaster+4 -4
mediawiki/extensions/UploadWizardmaster+2 -2
mediawiki/extensions/Wikibasemaster+40 -40
mediawiki/extensions/ContentTranslationmaster+2 -2
mediawiki/extensions/MultimediaViewermaster+10 -10
mediawiki/extensions/MultimediaViewermaster+9 -3
mediawiki/extensions/CentralNoticemaster+6 -6
mediawiki/extensions/CentralAuthmaster+2 -2
mediawiki/extensions/Echomaster+2 -2
mediawiki/extensions/Translatemaster+27 -35
mediawiki/extensions/Wikispeechmaster+8 -13
mediawiki/extensions/GuidedTourmaster+2 -2
mediawiki/extensions/WikiEditormaster+10 -12
mediawiki/extensions/Thanksmaster+25 -26
mediawiki/coremaster+108 -140
mediawiki/coremaster+18 -44
mediawiki/coremaster+7 -6
mediawiki/coremaster+106 -62
mediawiki/coremaster+52 -5
mediawiki/coremaster+16 -34
mediawiki/coremaster+1 -0
mediawiki/coremaster+31 -5
mediawiki/coremaster+338 -39
mediawiki/coremaster+18 -16
mediawiki/coremaster+2 -2
mediawiki/coremaster+6 -9
mediawiki/coremaster+5 -3
mediawiki/coremaster+10 -9
mediawiki/coremaster+1 K -679
mediawiki/coremaster+2 -5
mediawiki/coremaster+32 -66
mediawiki/coremaster+6 -22
mediawiki/coremaster+13 -24
mediawiki/extensions/WikibaseLexememaster+0 -1
mediawiki/extensions/Wikibasemaster+148 -216
Show related patches Customize query in gerrit

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Change 592090 merged by jenkins-bot:
[mediawiki/extensions/WikibaseLexeme@master] qunit: Remove unused 'wikibase.tests.qunit.testrunner' dependency

https://gerrit.wikimedia.org/r/592090

Change 592055 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] qunit: Remove QUnit.newWbEnvironment() abstraction layer

https://gerrit.wikimedia.org/r/592055

Change 592039 merged by jenkins-bot:
[mediawiki/core@master] qunit: Use "test.*" module names only for actual test suites

https://gerrit.wikimedia.org/r/592039

Change 592044 merged by jenkins-bot:
[mediawiki/core@master] qunit: Load qunitjs directly instead of via load.php

https://gerrit.wikimedia.org/r/592044

Change 592046 merged by jenkins-bot:
[mediawiki/core@master] qunit: Remove div#mw-javascripttest-summary from SpecialJavaScriptTest

https://gerrit.wikimedia.org/r/592046

Change 592050 merged by jenkins-bot:
[mediawiki/core@master] qunit: Remove dead code from SpecialJavaScriptTest

https://gerrit.wikimedia.org/r/592050

Change 592118 merged by jenkins-bot:
[mediawiki/core@master] qunit: Disable mw.loader.store server-side instead of client-side

https://gerrit.wikimedia.org/r/592118

Change 592083 merged by jenkins-bot:
[mediawiki/core@master] mediawiki.jqueryMsg: Refactor test suite to not make any API requests

https://gerrit.wikimedia.org/r/592083

Change 591988 merged by jenkins-bot:
[mediawiki/core@master] resourceloader: Fix load.mock.php query parameter corruption in tests

https://gerrit.wikimedia.org/r/591988

Change 595331 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[mediawiki/core@master] mediawiki.base: Let exists() work even in lang=qqx

https://gerrit.wikimedia.org/r/595331

Change 592143 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[mediawiki/core@master] qunit: Load JS with lang=qqx/skin=fallback as context

https://gerrit.wikimedia.org/r/592143

Change 595331 merged by jenkins-bot:
[mediawiki/core@master] mediawiki.base: Let exists() work even in lang=qqx

https://gerrit.wikimedia.org/r/595331

Change 592143 merged by jenkins-bot:
[mediawiki/core@master] qunit: Load JS with lang=qqx/skin=fallback as context

https://gerrit.wikimedia.org/r/592143

Change 744072 had a related patch set uploaded (by Krinkle; author: Krinkle):

[mediawiki/core@master] resourceloader: Allow debug=2 on JavaScriptTest, and misc whitespace

https://gerrit.wikimedia.org/r/744072

Change 744072 merged by jenkins-bot:

[mediawiki/core@master] resourceloader: Allow debug=2 on JavaScriptTest, and misc whitespace

https://gerrit.wikimedia.org/r/744072

Change 759209 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[mediawiki/core@master] QUnit: Allow running tests for a single skin/extension

https://gerrit.wikimedia.org/r/759209

Change 759215 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[mediawiki/core@master] QUnit: Set default file type to js

https://gerrit.wikimedia.org/r/759215

Change 759239 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[mediawiki/core@master] [WIP] QUnit: CLI option for watching test and source files

https://gerrit.wikimedia.org/r/759239

Change 759705 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[mediawiki/core@master] [WIP] QUnit: Add support for generating coverage reports

https://gerrit.wikimedia.org/r/759705

Change 759209 merged by jenkins-bot:

[mediawiki/core@master] QUnit: Allow running tests for core or single skin/extension

https://gerrit.wikimedia.org/r/759209

Change 759215 merged by jenkins-bot:

[mediawiki/core@master] QUnit: Set default file type to js

https://gerrit.wikimedia.org/r/759215

Change 763226 had a related patch set uploaded (by Krinkle; author: Krinkle):

[mediawiki/core@master] resourceloader: Replace QUnit.fixurl with mocked no-cache instruction

https://gerrit.wikimedia.org/r/763226

Change 763226 merged by jenkins-bot:

[mediawiki/core@master] resourceloader: Replace QUnit.fixurl with mocked no-cache instruction

https://gerrit.wikimedia.org/r/763226

Change 786427 had a related patch set uploaded (by Krinkle; author: Krinkle):

[mediawiki/core@master] qunit: Use inline variables and improve various doc blocks

https://gerrit.wikimedia.org/r/786427

Change 786428 had a related patch set uploaded (by Krinkle; author: Krinkle):

[mediawiki/core@master] qunit: Simplify mw.config and mw.messages resets

https://gerrit.wikimedia.org/r/786428

Change 786429 had a related patch set uploaded (by Krinkle; author: Krinkle):

[mediawiki/core@master] qunit: Remove redundant fixture reset

https://gerrit.wikimedia.org/r/786429

Change 786433 had a related patch set uploaded (by Krinkle; author: Krinkle):

[mediawiki/core@master] qunit: Clean up meta unit tests for testrunner.js

https://gerrit.wikimedia.org/r/786433

Change 786427 merged by jenkins-bot:

[mediawiki/core@master] qunit: Use inline variables and improve various doc blocks

https://gerrit.wikimedia.org/r/786427

Change 786428 merged by jenkins-bot:

[mediawiki/core@master] qunit: Simplify mw.config and mw.messages resets

https://gerrit.wikimedia.org/r/786428

Change 786429 merged by jenkins-bot:

[mediawiki/core@master] qunit: Remove redundant fixture reset

https://gerrit.wikimedia.org/r/786429

Change 786433 merged by jenkins-bot:

[mediawiki/core@master] qunit: Clean up meta unit tests for testrunner.js

https://gerrit.wikimedia.org/r/786433

Change 788443 had a related patch set uploaded (by Krinkle; author: Krinkle):

[mediawiki/extensions/WikiEditor@master] tests: Remove use of QUnit 1.x setup()/teardown()

https://gerrit.wikimedia.org/r/788443

Change 788444 had a related patch set uploaded (by Krinkle; author: Krinkle):

[mediawiki/extensions/Translate@master] tests: Remove use of QUnit 1.x setup()/teardown()

https://gerrit.wikimedia.org/r/788444

Change 789099 had a related patch set uploaded (by Krinkle; author: Krinkle):

[mediawiki/extensions/Thanks@master] tests: Remove use of QUnit 1.x setup()/teardown()

https://gerrit.wikimedia.org/r/789099

Change 788443 merged by jenkins-bot:

[mediawiki/extensions/WikiEditor@master] tests: Remove use of QUnit 1.x setup()/teardown()

https://gerrit.wikimedia.org/r/788443

Change 789099 merged by jenkins-bot:

[mediawiki/extensions/Thanks@master] tests: Remove use of QUnit 1.x setup()/teardown()

https://gerrit.wikimedia.org/r/789099

Change 789133 had a related patch set uploaded (by Krinkle; author: Krinkle):

[mediawiki/extensions/Wikispeech@master] tests: Remove use of QUnit 1.x setup()/teardown()

https://gerrit.wikimedia.org/r/789133

Change 789155 had a related patch set uploaded (by Krinkle; author: Krinkle):

[mediawiki/extensions/GuidedTour@master] tests: Remove use of QUnit 1.x setup()/teardown()

https://gerrit.wikimedia.org/r/789155

Change 789133 merged by jenkins-bot:

[mediawiki/extensions/Wikispeech@master] tests: Remove use of QUnit 1.x setup()/teardown()

https://gerrit.wikimedia.org/r/789133

Change 789155 merged by jenkins-bot:

[mediawiki/extensions/GuidedTour@master] tests: Remove use of QUnit 1.x setup()/teardown()

https://gerrit.wikimedia.org/r/789155

Change 788444 merged by jenkins-bot:

[mediawiki/extensions/Translate@master] tests: Remove use of QUnit 1.x setup()/teardown()

https://gerrit.wikimedia.org/r/788444

Change 791645 had a related patch set uploaded (by Krinkle; author: Krinkle):

[mediawiki/extensions/Echo@master] tests: Remove use of QUnit 1.x setup()/teardown()

https://gerrit.wikimedia.org/r/791645

Change 791657 had a related patch set uploaded (by Krinkle; author: Krinkle):

[mediawiki/extensions/Wikibase@master] tests: Remove use of QUnit 1.x setup()/teardown()

https://gerrit.wikimedia.org/r/791657

Change 791658 had a related patch set uploaded (by Krinkle; author: Krinkle):

[mediawiki/extensions/CentralAuth@master] tests: Remove use of QUnit 1.x setup()/teardown()

https://gerrit.wikimedia.org/r/791658

Change 791661 had a related patch set uploaded (by Krinkle; author: Krinkle):

[mediawiki/extensions/ContentTranslation@master] tests: Remove use of QUnit 1.x setup()/teardown()

https://gerrit.wikimedia.org/r/791661

Change 791664 had a related patch set uploaded (by Krinkle; author: Krinkle):

[mediawiki/extensions/CentralNotice@master] tests: Remove use of QUnit 1.x setup()/teardown()

https://gerrit.wikimedia.org/r/791664

Change 791645 merged by jenkins-bot:

[mediawiki/extensions/Echo@master] tests: Remove use of QUnit 1.x setup()/teardown()

https://gerrit.wikimedia.org/r/791645

Change 791658 merged by jenkins-bot:

[mediawiki/extensions/CentralAuth@master] tests: Remove use of QUnit 1.x setup()/teardown()

https://gerrit.wikimedia.org/r/791658

Change 791675 had a related patch set uploaded (by Krinkle; author: Krinkle):

[mediawiki/extensions/MultimediaViewer@master] tests: Remove use of QUnit 1.x setup()/teardown()

https://gerrit.wikimedia.org/r/791675

Change 791680 had a related patch set uploaded (by Krinkle; author: Krinkle):

[mediawiki/extensions/MultimediaViewer@master] tests: Make mmv.ui.metadataPanel.js pass with non-English locale

https://gerrit.wikimedia.org/r/791680

Change 791664 merged by jenkins-bot:

[mediawiki/extensions/CentralNotice@master] tests: Remove use of QUnit 1.x setup()/teardown()

https://gerrit.wikimedia.org/r/791664

Change 791675 merged by jenkins-bot:

[mediawiki/extensions/MultimediaViewer@master] tests: Remove use of QUnit 1.x setup()/teardown()

https://gerrit.wikimedia.org/r/791675

Change 791680 merged by jenkins-bot:

[mediawiki/extensions/MultimediaViewer@master] tests: Make mmv.ui.metadataPanel.js pass with non-English locale

https://gerrit.wikimedia.org/r/791680

Change 791661 merged by jenkins-bot:

[mediawiki/extensions/ContentTranslation@master] tests: Remove use of QUnit 1.x setup()/teardown()

https://gerrit.wikimedia.org/r/791661

Change 791657 merged by jenkins-bot:

[mediawiki/extensions/Wikibase@master] tests: Remove use of QUnit 1.x setup()/teardown()

https://gerrit.wikimedia.org/r/791657

Change 791756 had a related patch set uploaded (by Krinkle; author: Krinkle):

[mediawiki/extensions/UploadWizard@master] tests: Remove use of QUnit 1.x setup()/teardown()

https://gerrit.wikimedia.org/r/791756

Change 791756 merged by jenkins-bot:

[mediawiki/extensions/UploadWizard@master] tests: Remove use of QUnit 1.x setup()/teardown()

https://gerrit.wikimedia.org/r/791756

Change 791762 had a related patch set uploaded (by Krinkle; author: Krinkle):

[mediawiki/core@master] qunit: Remove use of QUnit 1.x setup()/teardown()

https://gerrit.wikimedia.org/r/791762

Change 791763 had a related patch set uploaded (by Krinkle; author: Krinkle):

[mediawiki/extensions/WikibaseLexeme@master] tests: Remove use of QUnit 1.x setup()/teardown()

https://gerrit.wikimedia.org/r/791763

Change 791764 had a related patch set uploaded (by Krinkle; author: Krinkle):

[mediawiki/skins/MinervaNeue@master] tests: Remove use of QUnit 1.x setup()/teardown()

https://gerrit.wikimedia.org/r/791764

Change 791764 merged by jenkins-bot:

[mediawiki/skins/MinervaNeue@master] tests: Remove use of QUnit 1.x setup()/teardown()

https://gerrit.wikimedia.org/r/791764

Change 791762 merged by jenkins-bot:

[mediawiki/core@master] qunit: Remove use of QUnit 1.x setup()/teardown()

https://gerrit.wikimedia.org/r/791762

Change 791763 merged by jenkins-bot:

[mediawiki/extensions/WikibaseLexeme@master] tests: Remove use of QUnit 1.x setup()/teardown()

https://gerrit.wikimedia.org/r/791763

Change 791732 had a related patch set uploaded (by Krinkle; author: Krinkle):

[mediawiki/extensions/CentralNotice@wmf_deploy] tests: Remove use of QUnit 1.x setup()/teardown()

https://gerrit.wikimedia.org/r/791732

Change 791732 merged by jenkins-bot:

[mediawiki/extensions/CentralNotice@wmf_deploy] tests: Remove use of QUnit 1.x setup()/teardown()

https://gerrit.wikimedia.org/r/791732

Change 791788 had a related patch set uploaded (by Krinkle; author: Krinkle):

[mediawiki/core@master] qunit: Remove custom QUnit 1.x setup()/teardown() aliases

https://gerrit.wikimedia.org/r/791788

Change 791789 had a related patch set uploaded (by Krinkle; author: Krinkle):

[mediawiki/core@master] qunit: Remove module() monkey-patch and use QUnit.hooks instead.

https://gerrit.wikimedia.org/r/791789

Change 791793 had a related patch set uploaded (by Krinkle; author: Krinkle):

[mediawiki/extensions/GrowthExperiments@master] tests: Remove use of QUnit 1.x setup()/teardown()

https://gerrit.wikimedia.org/r/791793

Change 791793 merged by jenkins-bot:

[mediawiki/extensions/GrowthExperiments@master] tests: Remove use of QUnit 1.x setup()/teardown()

https://gerrit.wikimedia.org/r/791793

Change 791788 merged by jenkins-bot:

[mediawiki/core@master] qunit: Remove custom QUnit 1.x setup()/teardown() aliases

https://gerrit.wikimedia.org/r/791788

Change 791789 merged by jenkins-bot:

[mediawiki/core@master] qunit: Remove module() monkey-patch and use QUnit.hooks instead.

https://gerrit.wikimedia.org/r/791789