Page MenuHomePhabricator

[EPIC] Speed up unit test execution and increase code coverage
Open, NormalPublic

Description

Documentation on wiki

https://mediawiki.org/wiki/Reading/Web/Projects/Invest_in_the_MobileFrontend_%26_MinervaNeue_frontend_architecture#Speed_up_unit_test_execution_and_increase_code_coverage

What

Tests are a crucial part in the development flow to verify that the code works as intended and is future-proof against changes of behavior. With the current stack tests take a long time to run (because of the ResourceLoader strain and the overhead of running through MediaWiki). As a result of the cost of running tests and the slow feedback cycle during the development flow, we haven’t kept up with properly testing the code, and this has negative effects on feature stability and resilience, and on confidence in changing the code.

Leveraging the Automatic bundling goal, we will follow the Extension:Popups approach and migrate as many unit tests as we easily can to run in the CLI (Node.js based). Tests that make extensive use of MediaWiki or are too tied to ResourceLoader will be considered integration tests and kept in mediawiki-qunit.

Untested files simple enough to test will have new tests added in the process of migration for Automatic bundling.

Progress

Progress can be measured by looking at the amount of files migrated to the src/ folder, like in the previous sub-project (automatic bundling), and the amount of test files moved or left from tests/qunit to tests/node-qunit.

Currently, we have low test coverage and are unable to accurately measure test coverage. Test coverage will be measured using istanbul as files are migrated to node-qunit and we’d aim for ~75% coverage.

Measuring progress of increasing code coverage

"speeding up unit test execution" gave us the power to measure code coverage from the command line. We can now monitor progress much more carefully by looking at overall code coverage. This is documented in:
https://www.mediawiki.org/wiki/Reading/Web/Projects/Invest_in_the_MobileFrontend_%26_MinervaNeue_frontend_architecture/Progress?useskin=vector#Goal:_Increasing_code_coverage

At the end of the project we will demonstrate which components were poorly tested but are no longer in

Evaluating success

We will measure which files are tested, and how long it takes to run the tests in the current setup (page load + text execution), and compare it with the files tested at the end, and the test execution time (cli run + test execution). We will compare both cold start, and “dev” mode (watching files to re-run tests as code changes).

We will be successful if we have more files tested, we are measuring test coverage, and the tests run faster.

We will monitor number of client side errors. As we add unit tests, we hope to see the number of errors decrease as we close out edge cases.

AC

  • We have a measure before the project starts about how many files are tested, and how many tests are run, and how long they take to run in some specific machine (we’ll have to use the same machine at the end of the project to measure the new setup)
  • A node.js qunit script is added to the npm scripts under test:node (Using the qunit package either directly or via mw-node-qunit)
  • The node.js tests run on CI per commit via the npm test job
  • The existing browser qunit tests are migrated to tests/node-qunit (where it makes sense) and run under Node.js, using ES modules (see Extension:Popups)
  • There is a coverage npm script that when run generates test coverage reports
  • We have measured the new setup and compared the info with the old setup, and reported on success metrics

Related Objects

StatusAssignedTask
OpenNone
OpenNone
ResolvedJdlrobson
ResolvedJdrewniak
Resolvedpmiazga
ResolvedJdlrobson
ResolvedNone
ResolvedNone
ResolvedJdlrobson
ResolvedJdrewniak
ResolvedJdlrobson
ResolvedNone
ResolvedNone
Resolvedphuedx
ResolvedJdlrobson
ResolvedJdrewniak
ResolvedJdlrobson
OpenNone
OpenNone
StalledNone
OpenNone
OpenNone
OpenNone
ResolvedKrinkle
Resolvedovasileva
ResolvedJdlrobson
OpenNone
ResolvedNone
ResolvedNone
ResolvedNiedzielski
Resolvedovasileva
OpenJdlrobson
ResolvedJdlrobson
Resolvedphuedx
OpenNone
Resolvedovasileva
DeclinedJdlrobson
Resolvednray
StalledNone
Openovasileva
ResolvedNone
OpenNone
OpenNone
DuplicateNone
OpenNone
Resolvedpmiazga
ResolvedJdlrobson
DeclinedNone
OpenNone
ResolvedJdlrobson
DeclinedNone
ResolvedNiedzielski
DuplicateReedy
ResolvedNiedzielski
ResolvedJdlrobson
ResolvedJdlrobson
DuplicateNone
Resolvedpmiazga
Resolvedovasileva
Resolvedovasileva
Resolvedovasileva
ResolvedNiedzielski
OpenNone
OpenNone

Event Timeline

Test coverage possibly blocked on T193519.

ovasileva renamed this task from Speed up unit test execution and increase code coverage to [EPIC] Speed up unit test execution and increase code coverage.Jun 4 2018, 10:07 AM
Jdlrobson moved this task from Backlog to New adventures on the MinervaNeue board.Jun 13 2018, 3:55 PM
Vvjjkkii renamed this task from [EPIC] Speed up unit test execution and increase code coverage to ddcaaaaaaa.Jul 1 2018, 1:08 AM
Vvjjkkii triaged this task as High priority.
Vvjjkkii updated the task description. (Show Details)
Vvjjkkii removed a subscriber: Aklapper.
CommunityTechBot raised the priority of this task from High to Needs Triage.
CommunityTechBot renamed this task from ddcaaaaaaa to [EPIC] Speed up unit test execution and increase code coverage.
CommunityTechBot added a subscriber: Aklapper.
Jdlrobson triaged this task as High priority.Jul 9 2018, 4:29 PM
Jdlrobson moved this task from Inbox to Next up on the User-Jdlrobson board.Jul 31 2018, 1:58 AM

No subtasks remain! Time to check off the AC and close?

Nope. Very far away from resolving this one.
we have code coverage measuring now but we need to add test coverage for a bunch of things.

Will add some subtasks that are not associated.

Jdlrobson updated the task description. (Show Details)Feb 6 2019, 12:45 AM
Jdlrobson updated the task description. (Show Details)Feb 26 2019, 12:00 AM
Jdlrobson raised the priority of this task from High to Needs Triage.
Jdlrobson moved this task from Next up to Inbox on the User-Jdlrobson board.Feb 28 2019, 8:59 PM
Jdlrobson triaged this task as Normal priority.Mar 29 2019, 3:15 AM