Change 154757 had a related patch set uploaded by Legoktm:
Add code coverage template for extensions and try to add one for Echo

https://gerrit.wikimedia.org/r/154757

I set up some stuff on labs that can run them in the meantime: https://tools.wmflabs.org/coverage/

Kunal did a nice first pass! Poke me to refine the job and learn how to deploy + test it out :]

@Legoktm: Can we get MobileFrontend and Gather added to the list of extensions that you're generating reports for?

In T71685#1328430, @phuedx wrote:

@Legoktm: Can we get MobileFrontend and Gather added to the list of extensions that you're generating reports for?

So I generated https://tools.wmflabs.org/coverage/MobileFrontend/extensions_MobileFrontend.html ...and then realized that MF has no @covers tags...without them (see https://phpunit.de/manual/3.7/en/appendixes.annotations.html) the code coverage report is useless.

My current setup doesn't support extension dependencies, I can probably add that in later for Gather.

@Legoktm: Can we add math here https://tools.wmflabs.org/coverage/ ?

FYI @hashar I've begun adding some tags here - https://gerrit.wikimedia.org/r/276899
Am happy to continue and clean up the rest once I've got confirmation that you guys are still interested in autogenerating coverage.

Does that mean no?

Patches that enable generating coverage reports and publishing it post-merge are definitively welcome. Does that answer your question?

Yeh sorry, my question could have been clearer.
I guess my question is will https://gerrit.wikimedia.org/r/276899 address https://phabricator.wikimedia.org/T71685#1332687 or were the MobileFrontend reports disabled?

Yes it will.

@JanZerebecki that said, we have @cover tags in the repository but they are not showing up (see https://phabricator.wikimedia.org/T102006#2141553) how can I get that to update?

Best by fixing this task, as https://tools.wmflabs.org/coverage/ didn't update since somewhen in 2015, see the bottom right on a coverage report for its generation date.

Thanks @JanZerebecki

Change 279196 had a related patch set uploaded (by Jdlrobson):
Publish MobileFrontend phpunit test coverage

https://gerrit.wikimedia.org/r/279196

Change 279196 abandoned by Jdlrobson:
Publish MobileFrontend phpunit test coverage

Reason:
Doh. outside my area of expertise i'm afraid :(

https://gerrit.wikimedia.org/r/279196

The main issue that prevents this from not working right now is our <whitelist addUncoveredFilesFromWhitelist="true"> setting, that does not include extensions (or skins). Adding those unconditionally is most likely problematic from a performance perspective (c.f. 6aa1b85954192d4e13850ec8b53ed9bea7428480). Also, we don't want to whitelist includes/maintenance/etc from core because those are irrelevant here.

My current idea/plan is to have a script that runs as part of the extension coverage job, which modifies suite.xml to have something like the following:

		<whitelist addUncoveredFilesFromWhitelist="true">
			<directory suffix=".php">../../extensions/MassMessage/includes</directory>
			<directory suffix=".php">../../extensions/MassMessage/src</directory>
			<directory suffix=".php">../../extensions/MassMessage/maintenance</directory>
		</whitelist>

This would require extensions to have their PHP code in those 3 directories (if they want usable coverage reports), which I think is reasonable given we're already going in the PSR-4 direction. Whitelisting all of extensions/ won't work since we don't want to include coverage of extension dependencies. We could whitelist the entire extension directory, and blacklist common directories like tests/ and vendor/ but PHPUnit recommends using the whitelist feature over blacklisting.

Then we run the normal phpunit.php --testsuite extensions --coverage-html ... --coverage-clover ... and sync the coverage to doc.wm.o.

Change 401657 had a related patch set uploaded (by Legoktm; owner: Legoktm):
[integration/jenkins@master] Add suite-edit.py

https://gerrit.wikimedia.org/r/401657

Change 401657 merged by jenkins-bot:
[integration/jenkins@master] Add suite-edit.py

https://gerrit.wikimedia.org/r/401657

Change 401662 had a related patch set uploaded (by Legoktm; owner: Legoktm):
[integration/config@master] Add mwext-phpunit-coverage-publish job

https://gerrit.wikimedia.org/r/401662

Change 401662 merged by jenkins-bot:
[integration/config@master] Add mwext-phpunit-coverage-publish job

https://gerrit.wikimedia.org/r/401662

Ta-da: https://doc.wikimedia.org/cover/extensions/Linter/ - still have some bugs to figure out, but the PoC works.

OK, I also added this for https://doc.wikimedia.org/cover/extensions/ORES/

These reports are generated with PHP 7+xdebug, and run post-merge. Only one job runs at a time, so it might take a while. Extensions must have their source code in src or includes, and set up PHPUnit tests with @covers tags.

Please comment if you want your repo(s) added as part of the beta-ish period and we can try how well it performs.

@Legoktm

In T71685#2107621, @Physikerwelt wrote:

@Legoktm: Can we add math here https://tools.wmflabs.org/coverage/ ?

In T71685#3870865, @Physikerwelt wrote:

@Legoktm

In T71685#2107621, @Physikerwelt wrote:

@Legoktm: Can we add math here https://tools.wmflabs.org/coverage/ ?

Most (all?) of the PHP code is currently in the root directory, it needs to be in includes/ (or src/).

In T71685#3870880, @Legoktm wrote:

Most (all?) of the PHP code is currently in the root directory, it needs to be in includes/ (or src/).

I deployed rCICF867db16b0a61: Math: Run mwext-phpunit-coverage-publish as postmerge, so once a patch to Math is merged that moves the PHP files, coverage will start showing up.

Change 401756 had a related patch set uploaded (by Ladsgroup; owner: Amir Sarabadani):
[mediawiki/extensions/ORES@master] Add missing covers tags

https://gerrit.wikimedia.org/r/401756

Change 401756 merged by jenkins-bot:
[mediawiki/extensions/ORES@master] Add missing covers tags

https://gerrit.wikimedia.org/r/401756

Coverage reports generated so far: https://doc.wikimedia.org/cover/extensions/

The one for Math ( https://doc.wikimedia.org/cover/extensions/Math/ ) the main page starts at /maintenance but the others reports looks fine.

Well done @Legoktm !

I ended up moving the extension reports to https://doc.wikimedia.org/cover-extensions/ (dash, not slash) because the PHP code powering that site isn't really set up to handle subdirectories.

https://doc.wikimedia.org/cover/ now has a breadcrumb navigation link to extension coverage reports and back. It also has a quick summary explaining the concept of test coverage if you're not already familiar.

The coverage job now only runs for commits to master and includes all of the extension dependencies. It also uses PHP 7.0 which doesn't have all of the different extensions installed yet, and measures coverage differently from 5.5. But it's also much faster :)

Hmm. Is this meant to be both PHP and JS? Would love it to be both but first steps first?

Just PHPUnit. T116808: Provide a CI job to generate JS code coverage reports for extensions by using karma-coverage instead of just karma tracks JavaScript coverage reports for extensions. Thanks for fixing the title :)

In T71685#3870646, @Legoktm wrote:

Please comment if you want your repo(s) added as part of the beta-ish period and we can try how well it performs.

The WikibaseQualityConstraints extension has @covers tags, and its source code resides in src/ (except for a maintenance/ entry point without tests, but that only delegates to other classes in src/), so if I understand this task correctly it could qualify.

Edit: now part of T185211: Generate coverage report for Wikidata extensions (cc @Ladsgroup)