Page MenuHomePhabricator

Generate PHP code coverage reports for extensions
Closed, ResolvedPublic

Tokens
"Like" token, awarded by Physikerwelt."Mountain of Wealth" token, awarded by D3r1ck01."Like" token, awarded by Ladsgroup."Doubloon" token, awarded by Ricordisamoa."Like" token, awarded by phuedx."Manufacturing Defect?" token, awarded by Addshore.
Assigned To
Authored By
Legoktm, Aug 18 2014

Description

core has a coverage report at https://integration.wikimedia.org/cover/mediawiki-core/master/php/, it would be nice if we could also generate nightly reports for extensions too.

Details

Reference
bz69685
Related Gerrit Patches:
mediawiki/extensions/ORES : masterAdd missing covers tags
integration/config : masterAdd mwext-phpunit-coverage-publish job
integration/jenkins : masterAdd suite-edit.py
integration/config : masterPublish MobileFrontend phpunit test coverage

Related Objects

Event Timeline

bzimport raised the priority of this task from to Medium.Nov 22 2014, 3:39 AM
bzimport set Reference to bz69685.
bzimport added a subscriber: Unknown Object (MLST).
Legoktm created this task.Aug 18 2014, 6:13 AM

Change 154757 had a related patch set uploaded by Legoktm:
Add code coverage template for extensions and try to add one for Echo

https://gerrit.wikimedia.org/r/154757

I set up some stuff on labs that can run them in the meantime: https://tools.wmflabs.org/coverage/

Kunal did a nice first pass! Poke me to refine the job and learn how to deploy + test it out :]

phuedx added a subscriber: phuedx.Jun 2 2015, 9:31 AM

@Legoktm: Can we get MobileFrontend and Gather added to the list of extensions that you're generating reports for?

phuedx awarded a token.Jun 2 2015, 5:05 PM

@Legoktm: Can we get MobileFrontend and Gather added to the list of extensions that you're generating reports for?

So I generated https://tools.wmflabs.org/coverage/MobileFrontend/extensions_MobileFrontend.html ...and then realized that MF has no @covers tags...without them (see https://phpunit.de/manual/3.7/en/appendixes.annotations.html) the code coverage report is useless.

My current setup doesn't support extension dependencies, I can probably add that in later for Gather.

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 4 2016, 4:01 AM
Krinkle removed a subscriber: Krinkle.Feb 23 2016, 4:53 PM

FYI @hashar I've begun adding some tags here - https://gerrit.wikimedia.org/r/276899
Am happy to continue and clean up the rest once I've got confirmation that you guys are still interested in autogenerating coverage.

Does that mean no?

Patches that enable generating coverage reports and publishing it post-merge are definitively welcome. Does that answer your question?

Yeh sorry, my question could have been clearer.
I guess my question is will https://gerrit.wikimedia.org/r/276899 address https://phabricator.wikimedia.org/T71685#1332687 or were the MobileFrontend reports disabled?

@JanZerebecki that said, we have @cover tags in the repository but they are not showing up (see https://phabricator.wikimedia.org/T102006#2141553) how can I get that to update?

Best by fixing this task, as https://tools.wmflabs.org/coverage/ didn't update since somewhen in 2015, see the bottom right on a coverage report for its generation date.

Change 279196 had a related patch set uploaded (by Jdlrobson):
Publish MobileFrontend phpunit test coverage

https://gerrit.wikimedia.org/r/279196

Change 279196 abandoned by Jdlrobson:
Publish MobileFrontend phpunit test coverage

Reason:
Doh. outside my area of expertise i'm afraid :(

https://gerrit.wikimedia.org/r/279196

D3r1ck01 added a subscriber: D3r1ck01.

The main issue that prevents this from not working right now is our <whitelist addUncoveredFilesFromWhitelist="true"> setting, that does not include extensions (or skins). Adding those unconditionally is most likely problematic from a performance perspective (c.f. 6aa1b85954192d4e13850ec8b53ed9bea7428480). Also, we don't want to whitelist includes/maintenance/etc from core because those are irrelevant here.

My current idea/plan is to have a script that runs as part of the extension coverage job, which modifies suite.xml to have something like the following:

		<whitelist addUncoveredFilesFromWhitelist="true">
			<directory suffix=".php">../../extensions/MassMessage/includes</directory>
			<directory suffix=".php">../../extensions/MassMessage/src</directory>
			<directory suffix=".php">../../extensions/MassMessage/maintenance</directory>
		</whitelist>

This would require extensions to have their PHP code in those 3 directories (if they want usable coverage reports), which I think is reasonable given we're already going in the PSR-4 direction. Whitelisting all of extensions/ won't work since we don't want to include coverage of extension dependencies. We could whitelist the entire extension directory, and blacklist common directories like tests/ and vendor/ but PHPUnit recommends using the whitelist feature over blacklisting.

Then we run the normal phpunit.php --testsuite extensions --coverage-html ... --coverage-clover ... and sync the coverage to doc.wm.o.

Legoktm claimed this task.Dec 29 2017, 8:48 PM

Change 401657 had a related patch set uploaded (by Legoktm; owner: Legoktm):
[integration/jenkins@master] Add suite-edit.py

https://gerrit.wikimedia.org/r/401657

Change 401657 merged by jenkins-bot:
[integration/jenkins@master] Add suite-edit.py

https://gerrit.wikimedia.org/r/401657

Change 401662 had a related patch set uploaded (by Legoktm; owner: Legoktm):
[integration/config@master] Add mwext-phpunit-coverage-publish job

https://gerrit.wikimedia.org/r/401662

Change 401662 merged by jenkins-bot:
[integration/config@master] Add mwext-phpunit-coverage-publish job

https://gerrit.wikimedia.org/r/401662

Legoktm added a comment.EditedJan 3 2018, 3:58 AM

Ta-da: https://doc.wikimedia.org/cover/extensions/Linter/ - still have some bugs to figure out, but the PoC works.

OK, I also added this for https://doc.wikimedia.org/cover/extensions/ORES/

These reports are generated with PHP 7+xdebug, and run post-merge. Only one job runs at a time, so it might take a while. Extensions must have their source code in src or includes, and set up PHPUnit tests with @covers tags.

Please comment if you want your repo(s) added as part of the beta-ish period and we can try how well it performs.

Most (all?) of the PHP code is currently in the root directory, it needs to be in includes/ (or src/).

Most (all?) of the PHP code is currently in the root directory, it needs to be in includes/ (or src/).

I deployed rCICF867db16b0a61: Math: Run mwext-phpunit-coverage-publish as postmerge, so once a patch to Math is merged that moves the PHP files, coverage will start showing up.

Change 401756 had a related patch set uploaded (by Ladsgroup; owner: Amir Sarabadani):
[mediawiki/extensions/ORES@master] Add missing covers tags

https://gerrit.wikimedia.org/r/401756

Change 401756 merged by jenkins-bot:
[mediawiki/extensions/ORES@master] Add missing covers tags

https://gerrit.wikimedia.org/r/401756

hashar added a comment.EditedJan 3 2018, 6:11 PM

Coverage reports generated so far: https://doc.wikimedia.org/cover/extensions/

The one for Math ( https://doc.wikimedia.org/cover/extensions/Math/ ) the main page starts at /maintenance but the others reports looks fine.

Well done @Legoktm !

I ended up moving the extension reports to https://doc.wikimedia.org/cover-extensions/ (dash, not slash) because the PHP code powering that site isn't really set up to handle subdirectories.

https://doc.wikimedia.org/cover/ now has a breadcrumb navigation link to extension coverage reports and back. It also has a quick summary explaining the concept of test coverage if you're not already familiar.

The coverage job now only runs for commits to master and includes all of the extension dependencies. It also uses PHP 7.0 which doesn't have all of the different extensions installed yet, and measures coverage differently from 5.5. But it's also much faster :)

Jdforrester-WMF renamed this task from Generate code coverage reports for extensions to Generate PHP code coverage reports for extensions.Jan 9 2018, 6:32 PM
Jdforrester-WMF updated the task description. (Show Details)
Jdforrester-WMF added a subscriber: Jdforrester-WMF.

Hmm. Is this meant to be both PHP and JS? Would love it to be both but first steps first?

Just PHPUnit. T116808: Provide a CI job to generate JS code coverage reports for extensions by using karma-coverage instead of just karma tracks JavaScript coverage reports for extensions. Thanks for fixing the title :)

Please comment if you want your repo(s) added as part of the beta-ish period and we can try how well it performs.

The WikibaseQualityConstraints extension has @covers tags, and its source code resides in src/ (except for a maintenance/ entry point without tests, but that only delegates to other classes in src/), so if I understand this task correctly it could qualify.

Edit: now part of T185211: Generate coverage report for Wikidata extensions (cc @Ladsgroup)

Legoktm closed this task as Resolved.Jan 23 2018, 11:42 PM

Announced: https://lists.wikimedia.org/pipermail/wikitech-l/2018-January/089437.html

Thanks everyone who helped test this, I think we can declare this resolved now :-)