Page MenuHomePhabricator

Use Jest in MediaWiki CI to test Vue.js components and plugins
Closed, ResolvedPublic

Description

Vue and Vuex have now been added to Core as resource modules. Now that we have basic support in place for using the framework, we should think about how we want to write tests for Vue components and plugins.

I'd like to propose that we use Jest to test our Vue code going forward. I know that we already have some other tools in place (QUnit and node-qunit) for testing JS code, but I believe that adding Jest to our pipeline would be useful for the following reasons:

  • We are writing single-file components without using Webpack, thanks to @Catrope's updates to ResourceLoader. Jest gives us the ability to test these files in a headless Node.js environment without needing to maintain a test-only build process thanks to the vue-jest library. This allows Jest to handle all transforms of Vue files itself without needing to depend on an additional tool.
  • Jest is a "batteries-included" system that brings its own assertions, mocking tools (Sinon isn't necessary when using it), etc.
  • Jest plays nicely with Vue's official testing tools, https://vue-test-utils.vuejs.org/

I have a working example of a Jest-based test suite for Vue components (and Vuex actions) here: https://github.com/egardner/mediawiki-extensions-MachineVision. In addition to the libraries themselves, here's what I had to add to get Jest, Vue, and MediaWiki working together:

  • Jest Config file specifying what needs to be tested, how to collect coverage reports, what files to transform using vue-jest, and what files to ignore
  • Jest setup file that mocks out a MediaWiki JS environment for us in testing. This is similar to what mw-node-qunit does. It may be worth pulling some of this code into a similar shared location.
  • A __mocks__ folder in the root project directory where module mocks can be defined (this works well when code requires a RL module that does not have an equivalent NPM package)
  • The tests themselves live in tests/jest and can be run with a simple jest command; this has been added to the test:unit package script in package.json

Adding similar scaffolding to MediaWiki core should be straightforward, we'd just need to ensure that tests intended for Jest are not run in node-qunit and vice-versa. @Krinkle @Catrope @Jdforrester-WMF thoughts on this?

Event Timeline

This sounds like a reasonable plan (though people will grumble about yet more stuff in the root of the repo). Happy to help with it.

@Jdforrester-WMF great, I can start a patch sometime this week and I'll be sure to add you.

That one is about the Web team's repo for their search widget (probably Vector), which won't need any special handling; this is about MW itself, which probably will.

Change #1068834 had a related patch set uploaded (by MusikAnimal; author: MusikAnimal):

[mediawiki/core@master] [PROOF OF CONCEPT] Add Jest JS testing lib to Core with an initial test

https://gerrit.wikimedia.org/r/1068834

POC now fully working. Huge thanks to @egardner for the help!

As of the time of writing, r1068834 would add 8 new NPM packages to MW Core. These are dev-only dependencies, and are already used in some extensions and in Codex. We can trim down the list some, if this is deemed preferable:

  • Codex and Vue for example could be mock-aliased to where they live in resources/lib/. This however might be fragile if files change directories, while LibUp could keep the versions of Codex/Vue in sync if it's in package.json.
  • We could bundle them all into a single MediaWiki-specific testing utility package, similar to https://github.com/wikimedia/mw-node-qunit/, and add only that to package.json

Other open questions:

  • Do the packages require security review?
  • Any concerns with this approach?

Speaking for Community-Tech, I feel getting something like Jest into Core is somewhat urgent as we are trying to wrap up the Multiblocks project. I don't feel comfortable shipping a critical Core feature without robust unit testing, and as per the description it seems existing tooling won't work well for our purposes.

Thanks for resurrecting this task! So far all Vue code has lived in skins or extensions, but if we are going to put components into Core directly then we should have a test suite there as well.

  • Codex and Vue for example could be mock-aliased to where they live in resources/lib/. This however might be fragile if files change directories, while LibUp could keep the versions of Codex/Vue in sync if it's in package.json.

Personally I would not recommend this – we already publish NPM packages of Codex et al which can be used in Node without problems – just add them as a dev dependency. We have LibUp to keep things in sync.

I think we'll primarily want to mock modules when our code is require()-ing RL modules that do not correspond to anything in package.json. The fact that Jest can do this easily (hence the name) was one of the original selling points of it IMO – this allows us to perform Node-based tests even though we use a custom module system in our front-end code.

I'm agnostic about this, but who would maintain such a package?

Other open questions:

  • Do the packages require security review?
  • Any concerns with this approach?

Speaking for Community-Tech, I feel getting something like Jest into Core is somewhat urgent as we are trying to wrap up the Multiblocks project. I don't feel comfortable shipping a critical Core feature without robust unit testing, and as per the description it seems existing tooling won't work well for our purposes.

Security review may be required here – I'd recommend reaching out to them. However there typically is a lead time of several months for new review requests. The fact that 1) this code is already being used elsewhere and 2) these are dev-only dependencies may mitigate this somewhat. Vector uses the same setup and it is typically shipped everywhere that MW core is. I'd still recommend checking though.

There's no security review required for Jest at this time as long as it remains a developer dependency. We would like to ensure that the version stays up to date and if there are any critical vulnerabilities found, that you reach out to the security team.

  • Codex and Vue for example could be mock-aliased to where they live in resources/lib/. This however might be fragile if files change directories, while LibUp could keep the versions of Codex/Vue in sync if it's in package.json.

Personally I would not recommend this – we already publish NPM packages of Codex et al which can be used in Node without problems – just add them as a dev dependency. We have LibUp to keep things in sync.

I think we'll primarily want to mock modules when our code is require()-ing RL modules that do not correspond to anything in package.json. The fact that Jest can do this easily (hence the name) was one of the original selling points of it IMO – this allows us to perform Node-based tests even though we use a custom module system in our front-end code.

+1, let's use the package.json dev-dep route here like in extensions.

I'm agnostic about this, but who would maintain such a package?

IME being outside of this repo makes for much more complex breakage/fix cycles. Let's maintain it here.

Other open questions:

  • Do the packages require security review?
  • Any concerns with this approach?

Speaking for Community-Tech, I feel getting something like Jest into Core is somewhat urgent as we are trying to wrap up the Multiblocks project. I don't feel comfortable shipping a critical Core feature without robust unit testing, and as per the description it seems existing tooling won't work well for our purposes.

Security review may be required here – I'd recommend reaching out to them. However there typically is a lead time of several months for new review requests. The fact that 1) this code is already being used elsewhere and 2) these are dev-only dependencies may mitigate this somewhat. Vector uses the same setup and it is typically shipped everywhere that MW core is. I'd still recommend checking though.

I'm happy to merge as-is given its widespread use. [ec] And Security say "go".

Awesome! I have removed the [Proof of concept] from the commit message. This is formally up for review now. Please do note this from the commit message:

The "overrides" added to package.json is a temporary measure to get around a current issue with caniuse-lite and the stylelint plugin that relies on it. See: https://github.com/RJWadley/stylelint-no-unsupported-browser-features/issues/299

I suspect this issue will be fixed relatively soon. I'm not sure if it's better to wait it out, or if we're comfortable locking caniuse-lite to a specific version temporarily.

FYI I only brought up ideas about reducing the packages under the belief there might be some pushback with so many new packages being added. I concur that it's preferable to have all individually in package.json.

Change #1068834 merged by jenkins-bot:

[mediawiki/core@master] Add Jest to MW Core for testing Vue components

https://gerrit.wikimedia.org/r/1068834

Change #1072309 had a related patch set uploaded (by Eric Gardner; author: Eric Gardner):

[mediawiki/core@master] Release notes: Add Jest test suite information

https://gerrit.wikimedia.org/r/1072309

Change #1072310 had a related patch set uploaded (by MusikAnimal; author: MusikAnimal):

[mediawiki/core@master] RELEASE-NOTES-1.43: document addition of Jest testing framework

https://gerrit.wikimedia.org/r/1072310

Change #1072310 abandoned by MusikAnimal:

[mediawiki/core@master] RELEASE-NOTES-1.43: document addition of Jest testing framework

Reason:

beaten by mere seconds I33dcc4aa91bdc8a4a4ac592bf93b8fa160225e55

https://gerrit.wikimedia.org/r/1072310

We are finalizing language for RELEASE-NOTES but this task can be considered complete. Thanks to all the participants for helping to close out this 4+ year old task!

Yes thank you all!

Should we do anything about coverage reports? It's be great to have them published on https://doc.wikimedia.org/cover/

I'm also working on the documentation. Please feel free to review/edit as you see fit! https://www.mediawiki.org/wiki/Jest

Yes thank you all!

Should we do anything about coverage reports? It's be great to have them published on https://doc.wikimedia.org/cover/

I'm also working on the documentation. Please feel free to review/edit as you see fit! https://www.mediawiki.org/wiki/Jest

Jest can produce a coverage report – just change the settings at the top of jest.config.js. By default it will write to a coverage directory in the project root, but again it can be configured. I was not sure how meaningful coverage is as a metric when different JS files might be covered by different test suites (Jest / QUnit / Special:JavascriptTest / etc) so I disabled it. But most other Jest suites we have set up do use this FWIW.

Should we do anything about coverage reports? It's be great to have them published on https://doc.wikimedia.org/cover/

Jest can produce a coverage report – just change the settings at the top of jest.config.js. By default it will write to a coverage directory in the project root, but again it can be configured. I was not sure how meaningful coverage is as a metric when different JS files might be covered by different test suites (Jest / QUnit / Special:JavascriptTest / etc) so I disabled it. But most other Jest suites we have set up do use this FWIW.

Indeed, I had coverage disabled in PS3 too, but mainly as I wasn't sure how to get it wired up to doc.wikimedia.org! Upon further investigation, it looks like the best we can do currently is to use SonarQube: https://sonarcloud.io/code?id=mediawiki-core&selected=mediawiki-core%3Aresources%2Fsrc It doesn't look like any JS coverage is being reported there, so I'm guessing it's safe for Jest to start logging to it?

Should we do anything about coverage reports? It's be great to have them published on https://doc.wikimedia.org/cover/

Jest can produce a coverage report – just change the settings at the top of jest.config.js. By default it will write to a coverage directory in the project root, but again it can be configured. I was not sure how meaningful coverage is as a metric when different JS files might be covered by different test suites (Jest / QUnit / Special:JavascriptTest / etc) so I disabled it. But most other Jest suites we have set up do use this FWIW.

Indeed, I had coverage disabled in PS3 too, but mainly as I wasn't sure how to get it wired up to doc.wikimedia.org! Upon further investigation, it looks like the best we can do currently is to use SonarQube: https://sonarcloud.io/code?id=mediawiki-core&selected=mediawiki-core%3Aresources%2Fsrc It doesn't look like any JS coverage is being reported there, so I'm guessing it's safe for Jest to start logging to it?

Yeah, it should be. FWIW that is what we did for GrowthExperiments as well: test coverage of Vue code is reported here https://sonarcloud.io/code?id=mediawiki-extensions-GrowthExperiments&selected=mediawiki-extensions-GrowthExperiments%3Amodules, see https://github.com/wikimedia/mediawiki-extensions-GrowthExperiments/blob/master/jest.config.js#L27

Change #1072636 had a related patch set uploaded (by VolkerE; author: VolkerE):

[design/codex@main] docs: Update RELEASING.md to include core Jest dependency updates

https://gerrit.wikimedia.org/r/1072636

Change #1072309 merged by jenkins-bot:

[mediawiki/core@master] Release notes: Add Jest test suite information

https://gerrit.wikimedia.org/r/1072309

Change #1072636 merged by jenkins-bot:

[design/codex@main] docs: Update RELEASING.md to include core Jest dependency updates

https://gerrit.wikimedia.org/r/1072636

Change #1073562 had a related patch set uploaded (by LWatson; author: LWatson):

[mediawiki/core@master] Update Codex from v1.12.0 to v1.13.0

https://gerrit.wikimedia.org/r/1073562

Change #1080382 had a related patch set uploaded (by MusikAnimal; author: MusikAnimal):

[mediawiki/core@master] package.json: add pinia and @pinia/testing

https://gerrit.wikimedia.org/r/1080382

Change #1080382 merged by jenkins-bot:

[mediawiki/core@master] package.json: add pinia and @pinia/testing

https://gerrit.wikimedia.org/r/1080382