Page MenuHomePhabricator
Create Task
Maniphest T378434

Document how to generate code coverage for extensions locally
Closed, ResolvedPublic
Actions

Description

Starting point. I found the following documentation to generate extension code coverage

https://www.mediawiki.org/wiki/Continuous_integration/Tutorials/Generating_PHP_test_coverage_for_a_MediaWiki_extension

in addition, I extracted that configuration from Jenkins

https://www.mediawiki.org/wiki/MediaWiki-Docker/Extension/Math#Code_coverage_report

which required MediaWiki-docker, which I would like to avoid, as running things locally feels more instant and reliable.
Q1 Is that even a good idea, or is everyone using mw-docker?

When trying to generate code-coverage reports locally, I ran into the following issues:

  • It took too long (I got a new computer)
  • I did need to set XDEBUG_MODE=coverage(It was not difficult to figure out, but could be added to the documentation.)
  • I added the following lines to suite.xml (almost as documented, not too difficult, but it makes the mw-core git repo dirty and harder to upgrade)
diff --git a/tests/phpunit/suite.xml b/tests/phpunit/suite.xml
index 85ed3369166..71b618ee946 100644
--- a/tests/phpunit/suite.xml
+++ b/tests/phpunit/suite.xml
@@ -72,6 +72,7 @@ This configuration file has been deprecated in favour of phpunit.xml.dist.
                <include>
                        <directory suffix=".php">../../includes</directory>
                        <directory suffix=".php">../../maintenance</directory>
+                       <directory suffix=".php">../../extensions/Math/src</directory>
  • The database tear-down failed as the service container had no configuration settings. I was unable to solve it and ignored the problem
diff --git a/tests/phpunit/MediaWikiIntegrationTestCase.php b/tests/phpunit/MediaWikiIntegrationTestCase.php
index 05873a58ba2..f40dd6ec955 100644
--- a/tests/phpunit/MediaWikiIntegrationTestCase.php
+++ b/tests/phpunit/MediaWikiIntegrationTestCase.php
@@ -1704,13 +1704,17 @@ abstract class MediaWikiIntegrationTestCase extends PHPUnit\Framework\TestCase {
 
                $services = MediaWikiServices::getInstance();
                ( new HookRunner( $services->getHookContainer() ) )->onUnitTestsBeforeDatabaseTeardown();
-
-               $jobQueueGroup = $services->getJobQueueGroup();
-               foreach ( $wgJobClasses as $type => $class ) {
-                       // Delete any jobs under the clone DB (or old prefix in other stores)
-                       $jobQueueGroup->get( $type )->delete();
+               try{
+                       $jobQueueGroup = $services->getJobQueueGroup();
+                       foreach ( $wgJobClasses as $type => $class ) {
+                               // Delete any jobs under the clone DB (or old prefix in other stores)
+                               $jobQueueGroup->get( $type )->delete();
+                       }
+               } catch (Exception $e) {
+                       // Ignore exceptions from the job queue, as they are not critical
                }
 
+
                if ( self::$dbClone ) {

without that, I see the following output (and no coverage report)

...                                                           2138 / 2138 (100%)PHP Fatal error:  Uncaught InvalidArgumentException: Key "JobTypeConf" not found in input sources in /Users/physikerwelt/git/mw/includes/config/ServiceOptions.php:60
Stack trace:
#0 /Users/physikerwelt/git/mw/includes/ServiceWiring.php(1049): MediaWiki\Config\ServiceOptions->__construct(Array, Object(MediaWiki\Config\GlobalVarConfig))
#1 /Users/physikerwelt/git/mw/vendor/wikimedia/services/src/ServiceContainer.php(445): Wikimedia\Services\ServiceContainer::{closure}(Object(MediaWiki\MediaWikiServices))
#2 /Users/physikerwelt/git/mw/vendor/wikimedia/services/src/ServiceContainer.php(411): Wikimedia\Services\ServiceContainer->createService('JobQueueGroupFa...')
#3 /Users/physikerwelt/git/mw/includes/MediaWikiServices.php(357): Wikimedia\Services\ServiceContainer->getService('JobQueueGroupFa...')
#4 /Users/physikerwelt/git/mw/includes/MediaWikiServices.php(1251): MediaWiki\MediaWikiServices->getService('JobQueueGroupFa...')
#5 /Users/physikerwelt/git/mw/includes/ServiceWiring.php(1044): MediaWiki\MediaWikiServices->getJobQueueGroupFactory()
#6 /Users/physikerwelt/git/mw/vendor/wikimedia/services/src/ServiceContainer.php(445): Wikimedia\Services\ServiceContainer::{closure}(Object(MediaWiki\MediaWikiServices))
#7 /Users/physikerwelt/git/mw/vendor/wikimedia/services/src/ServiceContainer.php(411): Wikimedia\Services\ServiceContainer->createService('JobQueueGroup')
#8 /Users/physikerwelt/git/mw/includes/MediaWikiServices.php(357): Wikimedia\Services\ServiceContainer->getService('JobQueueGroup')
#9 /Users/physikerwelt/git/mw/includes/MediaWikiServices.php(1244): MediaWiki\MediaWikiServices->getService('JobQueueGroup')
#10 /Users/physikerwelt/git/mw/tests/phpunit/MediaWikiIntegrationTestCase.php(1707): MediaWiki\MediaWikiServices->getJobQueueGroup()
#11 /Users/physikerwelt/git/mw/tests/phpunit/bootstrap.maintenance.php(28): MediaWikiIntegrationTestCase::teardownTestDB()
#12 [internal function]: PHPUnit\Util\FileLoader::{closure}()
#13 {main}
  thrown in /Users/physikerwelt/git/mw/includes/config/ServiceOptions.php on line 60
Fatal error: Uncaught InvalidArgumentException: Key "JobTypeConf" not found in input sources in /Users/physikerwelt/git/mw/includes/config/ServiceOptions.php:60
Stack trace:
#0 /Users/physikerwelt/git/mw/includes/ServiceWiring.php(1049): MediaWiki\Config\ServiceOptions->__construct(Array, Object(MediaWiki\Config\GlobalVarConfig))
#1 /Users/physikerwelt/git/mw/vendor/wikimedia/services/src/ServiceContainer.php(445): Wikimedia\Services\ServiceContainer::{closure}(Object(MediaWiki\MediaWikiServices))
#2 /Users/physikerwelt/git/mw/vendor/wikimedia/services/src/ServiceContainer.php(411): Wikimedia\Services\ServiceContainer->createService('JobQueueGroupFa...')
#3 /Users/physikerwelt/git/mw/includes/MediaWikiServices.php(357): Wikimedia\Services\ServiceContainer->getService('JobQueueGroupFa...')
#4 /Users/physikerwelt/git/mw/includes/MediaWikiServices.php(1251): MediaWiki\MediaWikiServices->getService('JobQueueGroupFa...')
#5 /Users/physikerwelt/git/mw/includes/ServiceWiring.php(1044): MediaWiki\MediaWikiServices->getJobQueueGroupFactory()
#6 /Users/physikerwelt/git/mw/vendor/wikimedia/services/src/ServiceContainer.php(445): Wikimedia\Services\ServiceContainer::{closure}(Object(MediaWiki\MediaWikiServices))
#7 /Users/physikerwelt/git/mw/vendor/wikimedia/services/src/ServiceContainer.php(411): Wikimedia\Services\ServiceContainer->createService('JobQueueGroup')
#8 /Users/physikerwelt/git/mw/includes/MediaWikiServices.php(357): Wikimedia\Services\ServiceContainer->getService('JobQueueGroup')
#9 /Users/physikerwelt/git/mw/includes/MediaWikiServices.php(1244): MediaWiki\MediaWikiServices->getService('JobQueueGroup')
#10 /Users/physikerwelt/git/mw/tests/phpunit/MediaWikiIntegrationTestCase.php(1707): MediaWiki\MediaWikiServices->getJobQueueGroup()
#11 /Users/physikerwelt/git/mw/tests/phpunit/bootstrap.maintenance.php(28): MediaWikiIntegrationTestCase::teardownTestDB()
#12 [internal function]: PHPUnit\Util\FileLoader::{closure}()
#13 {main}
  thrown in /Users/physikerwelt/git/mw/includes/config/ServiceOptions.php on line 60


You should really speed up these slow tests (>50ms)...
 1. 1005ms to run MediaWiki\\Extension\\Math\\Tests\\WikiTexVC\\AllTest::testRunCases with data set "Big"
 2. 478ms to run MediaWiki\\Extension\\Math\\Tests\\PreferencesIntegrationTest::testMathOptionRegistered
 3. 219ms to run MediaWiki\\Extension\\Math\\Tests\\WikiTexVC\\AllTest::testRunCases with data set "Literals (1)"
 4. 194ms to run MediaWiki\\Extension\\Math\\Tests\\WikiTexVC\\AllTest::testRunCases with data set "FUN_AR1"
 5. 145ms to run MediaWiki\\Extension\\Math\\Tests\\PreferencesIntegrationTest::testInvalidDefaultOptionFixed
 6. 141ms to run MediaWiki\\Extension\\Math\\Tests\\WikiTexVC\\AllTest::testRunCases with data set "Literals (3)"
 7. 122ms to run phpunit\\InputCheck\\InputCheckFactoryTest::testNewMathoidChecker
 8. 116ms to run MediaWiki\\Extension\\Math\\Tests\\WikiTexVC\\AllTest::testRunCases with data set "Matrices"
 9. 82ms to run MediaWiki\\Extension\\Math\\Tests\\WikiTexVC\\AllTest::testRunCases with data set "Delimiters (4)"
 10. 76ms to run MediaWiki\\Extension\\Math\\InputCheck\\LocalCheckerTest::testInvalid
...and there are 5 more above your threshold hidden from view
Key "JobTypeConf" not found in input sources

I'm not sure what the real reason for the error is. Maybe the Math extension runs database tests without properly using the desired interfaces, or I'm running the tests in the wrong way.

Details

Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
(DO NOT SUBMIT) repro for T378434Hasharmediawiki/extensions/Mathmaster+17 -27 K
Customize query in gerrit

Related Objects

Event Timeline

Physikerwelt created this task.Oct 29 2024, 5:24 AM
Restricted Application added a subscriber: • Aklapper. · View Herald TranscriptOct 29 2024, 5:24 AM
Michael subscribed.Nov 19 2024, 11:03 AM
bd808 subscribed.Nov 19 2024, 11:58 PM

I added the following lines to suite.xml (almost as documented, not too difficult, but it makes the mw-core git repo dirty and harder to upgrade)

That part is ugly, but it is also what happens in the CI job that generates the coverage report. The big difference being that for CI the edit of $MW_INSTALL_PATH/tests/phpunit/suite.xml is done in an ephemeral copy of the code base.

bd808 added a comment.Nov 20 2024, 12:41 AM

Key "JobTypeConf" not found in input sources seems to be a general problem when running the tests via composer phpunit:entrypoint -- ./extensions/Math/tests/phpunit/ unrelated to coverage reporting.

When I use composer phpunit:entrypoint -- --group Math to run the tests instead it appears that TestSetup::applyInitialConfig() runs and $wgJobTypeConf is populated as the tests expect.

bd808 added a comment.Nov 20 2024, 12:55 AM

I did need to set XDEBUG_MODE=coverage(It was not difficult to figure out, but could be added to the documentation.)

That envvar was in the docs until https://www.mediawiki.org/w/index.php?title=MediaWiki-Docker/Extension/Math&diff=next&oldid=5960520. I think the removal was accidental, so I put it back with https://www.mediawiki.org/w/index.php?title=MediaWiki-Docker/Extension/Math&diff=prev&oldid=6861163

• thcipriani subscribed.Nov 20 2024, 1:02 AM

Is the problem here that you'd like to not see that? Or that you'd like to generate the coverage report? Or both?

Commenting out the line you wrapped in a try block will generate the coverage report for me locally using:

mkdir -p ../coverage
XDEBUG_MODE=coverage php -dpcov.enable=1 -dpcov.directory=../coverage -d 'pcov.exclude=@(tests|vendor)@' vendor/bin/phpunit -v -c tests/phpunit/suite.xml ./extensions/Math/tests/phpunit/ --coverage-html ../coverage  --whitelist $(pwd)/extensions/Math

Or in mediawiki-docker using:

docker compose exec -e XDEBUG_MODE=coverage mediawiki php -dpcov.enable=1 -dpcov.directory=/var/www/html/w/extensions/Math -d 'pcov.exclude=@(tests|vendor)@' vendor/bin/phpunit -c tests/phpunit/suite.xml ./extensions/Math/tests/phpunit/ --coverage-html ./extensions/Math/tests/coverage  --whitelist /var/www/html/w/extensions/Math

As to why it's failing, it's something specific to the Math extension. I can recreate it locally. This has shown up for other service keys in the past, rarely, the reasons vary (see this search for other cases).

Also note that coverage is published by CI: https://doc.wikimedia.org/cover-extensions/Math/src/index.html

Hope that helps.

gerritbot added a comment.Nov 20 2024, 11:39 AM

Change #1093309 had a related patch set uploaded (by Hashar; author: Hashar):

[mediawiki/extensions/Math@master] (DO NOT SUBMIT) repro for T378434

https://gerrit.wikimedia.org/r/1093309

gerritbot added a project: Patch-For-Review.Nov 20 2024, 11:39 AM
• hashar added a project: MediaWiki-Core-JobQueue.EditedNov 20 2024, 11:47 AM
• hashar subscribed.

About the stacktrace, it is triggered when retrieving the JobQueueGroup service which has a service option based on wgJobTypeConf and that defaults to NULL and is thus not set.

MediaWikiIntegrationTestCase does set it to an in memory queue:

tests/phpunit/MediaWikiIntegrationTestCase.php
$overrides[ MainConfigNames::JobTypeConf ] = [ 'default' => [ 'class' => JobQueueMemory::class ] ];

But MediaWikiUnitTestCase does not.

I have send https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Math/+/1093309 to reproduce the problem which can then be run with php tests/phpunit/phpunit.php extensions/Math. There is something very weird happening such as an empty MediaWikiUnitTestCase to be present in order to reproduce the failure.

I have added MediaWiki-Core-JobQueue , but I honestly don't know what is happening :)

Physikerwelt added a comment.Nov 20 2024, 12:30 PM

I have been trying to find out which test causes the problem, but I could not identify the test.

#!/bin/bash
export XDEBUG_MODE=coverage
for filename in ./extensions/Math/tests/phpunit/*; do
	echo "$filename"
        php -dpcov.enable=1 -dpcov.directory=../coverage -d 'pcov.exclude=@(tests|vendor)@' vendor/bin/phpunit -v -c tests/phpunit/suite.xml "$filename" --coverage-html ../coverage  --whitelist $(pwd)/extensions/Math > $(basename "$filename".log)
done

Non of the test fails. However,

XDEBUG_MODE=coverage php -dpcov.enable=1 -dpcov.directory=../coverage -d 'pcov.exclude=@(tests|vendor)@' vendor/bin/phpunit -v -c tests/phpunit/suite.xml ./extensions/Math/tests/phpunit/ --coverage-html ../coverage  --whitelist $(pwd)/extensions/Math
Physikerwelt updated the task description. (Show Details)Nov 20 2024, 12:38 PM
Physikerwelt edited projects, added: Math; removed: Developer Productivity, Documentation.Nov 20 2024, 1:13 PM

Thank you for your help. I updated the documentation. Now the bug

  • The database tear-down failed as the service container had no configuration settings. I was unable to solve it and ignored the problem

remains open.

kostajh subscribed.Nov 21 2024, 9:31 AM

Two notes:

  • suite.xml is deprecated in favor of phpunit.xml.dist.
  • There is also a composer script in core that creates a phpunit.xml file (not in version control) for you, with the coverage paths already set. composer phpunit:coverage-edit -- extensions/Math. Then you'd run phpunit -c phpunit.xml
gerritbot added a comment.Dec 2 2024, 2:04 PM

Change #1093309 abandoned by Hashar:

[mediawiki/extensions/Math@master] (DO NOT SUBMIT) repro for T378434

https://gerrit.wikimedia.org/r/1093309

Maintenance_bot removed a project: Patch-For-Review.Dec 2 2024, 2:30 PM
Physikerwelt claimed this task.Feb 14 2025, 8:54 AM
Physikerwelt triaged this task as Low priority.
Physikerwelt added a project: MediaWiki-Documentation.
In T378434#10342998, @kostajh wrote:
  • There is also a composer script in core that creates a phpunit.xml file (not in version control) for you, with the coverage paths already set. composer phpunit:coverage-edit -- extensions/Math. Then you'd run phpunit -c phpunit.xml

@kostajh : I did the following:

~/git/mw git:
composer phpunit:coverage-edit -- extensions/Math
> MediaWiki\Composer\ComposerPhpunitXmlCoverageEdit::onEvent
~/git/mw git:[mathUnit]
phpunit -c phpunit.xml
zsh: command not found: phpunit

Could you please explain where your phpunit points? Is that the official phpunit or the phpunit from within Mediawiki?

When I run composer phpunit it runs many (63713) tests, and fails with

..........The following exception is caused by a process timeout
Check https://getcomposer.org/doc/06-config.md#process-timeout for details

In Process.php line 1205:
                                                              
  The process "phpunit" exceeded the timeout of 300 seconds.  
                                                              

phpunit [--dev] [--no-dev] [--] [<args>...]

However, I can run coverage from within PHPstorm myself, which executes

/opt/homebrew/bin/php -dxdebug.mode=coverage /Users/physikerwelt/git/mw/vendor/phpunit/phpunit/phpunit --coverage-clover /Users/physikerwelt/Library/Caches/JetBrains/PhpStorm2024.3/coverage/mw@WikiTexVC.xml --configuration /Users/physikerwelt/git/mw/phpunit.xml.dist /Users/physikerwelt/git/mw/extensions/Math/tests/phpunit/unit/WikiTexVC --teamcity

However, I want to get more people (i.e., students) on board to help develop MW (extensions). Thus, I would like to be sure that there is one canonical path to running (Math) extension tests alone.

Physikerwelt updated the task description. (Show Details)Feb 14 2025, 8:55 AM
Physikerwelt closed this task as Resolved.Feb 14 2025, 9:30 AM

I updated the description as well as I could. https://www.mediawiki.org/w/index.php?oldid=7020917 I think this is good enough, and the main problem (The database tear-down failed) vanished.

Physikerwelt mentioned this in T362091: mwext-phpunit-coverage-publish fails for Math.Mar 18 2026, 6:29 PM
Maintenance_bot added a project: MediaWiki-API-Platform-Team.Mar 18 2026, 6:30 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits