|Resolved||Jhernandez||T104318 [GOAL]: Testing practices & expose test coverage|
|Open||None||T100294 [EPIC] Encourage developers to increase code coverage|
|Resolved||Legoktm||T71685 Generate PHP code coverage reports for extensions|
|Stalled||None||T88434 generate Wikibase.git code coverage on Jenkins|
|Open||None||T74318 [Task] add possibility to generate proper phpunit coverage reports of Wikibase.git|
|Open||None||T88435 [Task] generate patch code coverage on gerrit patch-set upload for wikibase.git|
|Resolved||Jdlrobson||T102006 Add @covers annotations to MobileFrontend PHPUnit tests|
|Duplicate||None||T151333 Proof of concept to generate coverage report of some Wikimedia mediawiki extensions|
|Open||None||T185211 Generate coverage report for Wikidata extensions|
- Mentioned In
T184052: Add @covers tags for tests in WMDE Technical Wishes extensions and enable coverage reports
T151333: Proof of concept to generate coverage report of some Wikimedia mediawiki extensions
T88434: generate Wikibase.git code coverage on Jenkins
T116819: Cannot access gerrit change 154757
T100294: [EPIC] Encourage developers to increase code coverage
T108768: Create QA Health scoreboard
T102006: Add @covers annotations to MobileFrontend PHPUnit tests
- Mentioned Here
- T185211: Generate coverage report for Wikidata extensions
T116808: Provide a CI job to generate JS code coverage reports for extensions by using karma-coverage instead of just karma
rCICF867db16b0a61: Math: Run mwext-phpunit-coverage-publish as postmerge
rMW6aa1b8595419: phpunit: Remove skins/ from coverage index
So I generated https://tools.wmflabs.org/coverage/MobileFrontend/extensions_MobileFrontend.html ...and then realized that MF has no @covers tags...without them (see https://phpunit.de/manual/3.7/en/appendixes.annotations.html) the code coverage report is useless.
My current setup doesn't support extension dependencies, I can probably add that in later for Gather.
The main issue that prevents this from not working right now is our <whitelist addUncoveredFilesFromWhitelist="true"> setting, that does not include extensions (or skins). Adding those unconditionally is most likely problematic from a performance perspective (c.f. 6aa1b85954192d4e13850ec8b53ed9bea7428480). Also, we don't want to whitelist includes/maintenance/etc from core because those are irrelevant here.
My current idea/plan is to have a script that runs as part of the extension coverage job, which modifies suite.xml to have something like the following:
<whitelist addUncoveredFilesFromWhitelist="true"> <directory suffix=".php">../../extensions/MassMessage/includes</directory> <directory suffix=".php">../../extensions/MassMessage/src</directory> <directory suffix=".php">../../extensions/MassMessage/maintenance</directory> </whitelist>
This would require extensions to have their PHP code in those 3 directories (if they want usable coverage reports), which I think is reasonable given we're already going in the PSR-4 direction. Whitelisting all of extensions/ won't work since we don't want to include coverage of extension dependencies. We could whitelist the entire extension directory, and blacklist common directories like tests/ and vendor/ but PHPUnit recommends using the whitelist feature over blacklisting.
Then we run the normal phpunit.php --testsuite extensions --coverage-html ... --coverage-clover ... and sync the coverage to doc.wm.o.
OK, I also added this for https://doc.wikimedia.org/cover/extensions/ORES/
These reports are generated with PHP 7+xdebug, and run post-merge. Only one job runs at a time, so it might take a while. Extensions must have their source code in src or includes, and set up PHPUnit tests with @covers tags.
Please comment if you want your repo(s) added as part of the beta-ish period and we can try how well it performs.
I ended up moving the extension reports to https://doc.wikimedia.org/cover-extensions/ (dash, not slash) because the PHP code powering that site isn't really set up to handle subdirectories.
https://doc.wikimedia.org/cover/ now has a breadcrumb navigation link to extension coverage reports and back. It also has a quick summary explaining the concept of test coverage if you're not already familiar.
The coverage job now only runs for commits to master and includes all of the extension dependencies. It also uses PHP 7.0 which doesn't have all of the different extensions installed yet, and measures coverage differently from 5.5. But it's also much faster :)
The WikibaseQualityConstraints extension has @covers tags, and its source code resides in src/ (except for a maintenance/ entry point without tests, but that only delegates to other classes in src/), so if I understand this task correctly it could qualify.
Edit: now part of T185211: Generate coverage report for Wikidata extensions (cc @Ladsgroup)