Page MenuHomePhabricator

Proof of concept to generate coverage report of some Wikimedia mediawiki extensions
Closed, DuplicatePublic

Description

T71685 is quite old. There is interest in Wikimedia Reading team to enhance test coverage for the MediaWiki extensions they maintain.

We have the Jenkins job mediawiki-extensions-* that have a subset of Wikimedia deployed extensions. It should be fairly easy to copy paste the job and make it generate some coverage report on a daily basis. That would be a first good step.

Event Timeline

hashar created this task.Nov 22 2016, 3:41 PM
pmiazga added a subscriber: pmiazga.

@hashar any updates on this task? It would be awesome to see it live.

It is in the backlog of hundred of CI related tasks we have :-( Not quite sure how to tackle that one but here are some ideas.

What could help I guess is to figure out the exact command that would run tests for a given extension and generate the coverage report solely for that extension (filtering out MediaWiki, skin and extensions dependencies). Here is some context / brain dump:

RelatedArticles depends on BetaFeatures, Cards`, and MobileFrontend. The Jenkins job clones MediaWiki core, the extension and its dependencies, the workspace ends up being something like:

src
├── extensions/
│   ├── BetaFeatures/
│   ├── Cards/
│   ├── MobileFrontend/
│   └── RelatedArticles/
├── includes/
└── tests/
    └── phpunit/
        ├── phpunit.php
        └── suite.xml

Then we invoke:

cd src/tests/phpunit;  MW_INSTALL_PATH=src php phpunit.php --testsuite=extensions

That extensions suite is defined in MediaWiki core tests/phpunit/suite.xml:

tests/phpunit/suite.xml
<phpunit>
    <testsuites>
        <testsuite name="extensions">
            <directory>structure</directory>
            <file>suites/ExtensionsTestSuite.php</file>
            <file>suites/ExtensionsParserTestSuite.php</file>
            <file>suites/LessTestSuite.php</file>
        </testsuite>
...

So that ends up running:

  • MediaWiki core structure tests (such as the autoloader entries) against all extensions (not just RelatedArticles)
  • anything that:
    • got registered via the MediaWiki hooks UnitTestsList
    • discovered by globing tests/phpunit for each registered ext/skin
    • Parser tests registered by $wgParserTestFiles

I believe one can run solely RelatedArticles tests with php phpunit.php ../extensions/RelatedArticles. The test would execute mediawiki/core code and most probably dependent extensions as well. That would surely ends up in the coverage report which is not that useful(?).

So we would want to mess with PHPUnit coverage settings that have the ability to whitelist/exclude code path. Ideally we would only include in the coverage reports /extensions/RelatedArticles. The settings can't be passed to the phpunit command line, so we would need a phpunit XML configuration file for generate coverage of a single extension. https://phpunit.de/manual/4.8/en/code-coverage-analysis.html and specially Including and Excluding Files for Code Coverage.

One would want to craft a PHPUnit config file for that, and then in theory a CI job would "just" do:

cd tests/phpunit
php phpunit.php --configuration extension-coverage.xml --coverage-html "$WORKSPACE/log/coverage"

And publish to doc.wikimedia.org/cover the content of the resulting coverage file.

I believe one can run solely RelatedArticles tests with php phpunit.php ../extensions/RelatedArticles. The test would execute mediawiki/core code and most probably dependent extensions as well. That would surely ends up in the coverage report which is not that useful(?).

Why would that coverage report be not useful?
It would run the RelatedArticles tests and show you what code they covered, isn't that exactly what we want?