Changes and improvements to PHPUnit testing in MediaWiki

Building off the work done at the Prague Hackathon (T216260), we're happy to announce some significant changes and improvements to the PHP testing tools included with MediaWiki.

PHP unit tests can now be run statically, without installing MediaWiki

You can now download MediaWiki, run composer install, and then composer phpunit:unit to run core's unit test suite (T89432).

The standard PHPUnit entrypoint can be used, instead of the PHPUnit Maintenance class

You can now use the plain PHPUnit entrypoint at vendor/bin/phpunit instead of the MediaWiki maintenance class which wraps PHPUnit (tests/phpunit/phpunit.php).

Both the unit tests and integration tests can be executed with the standard phpunit entrypoint (vendor/bin/phpunit) or if you prefer, with the composer scripts defined in composer.json (e.g. composer phpunit:unit). We accomplished this by writing a new bootstrap.php file (the old one which the maintenance class uses was moved to tests/phpunit/bootstrap.maintenance.php) which executes the minimal amount of code necessary to make core, extension and skin classes discoverable by test classes.

Tests should be placed in tests/phpunit/{integration,unit}

Integration tests should be placed in tests/phpunit/integration while unit tests go in tests/phpunit/unit, these are discoverable by the new test suites (T87781). It sounds obvious now to write this, but a nice side effect is that by organizing tests into these directories it's immediately clear to authors and reviewers what type of test one is looking at.

Introducing MediaWikiUnitTestCase

A new base test case, MediaWikiUnitTestCase has been introduced with a minimal amount of boilerplate (@covers validator, ensuring the globals are disabled, and that the tests are in the proper directory, the default PHPUnit 4 and 6 compatibility layer). The MediaWikiTestCase has been renamed to MediaWikiIntegrationTestCase for clarity.

Please migrate tests to be unit tests where appropriate

A significant portion of core's unit tests have been ported to use MediaWikiUnitTestCase, approximately 50% of the total. We have also worked on porting extension tests to the unit/integration directories. @Ladsgroup wrote a helpful script to assist with automating the identification and moving of unit tests, see P8702. Migrating tests from MediaWikiIntegrationTestCase to MediaWikiUnitTestCase makes them faster.

Note that unit tests in CI are still run with the PHPUnit maintenance class (tests/phpunit/phpunit.php), so when reviewing unit test patches please execute them locally with vendor/bin/phpunit /path/to/tests/phpunit/unit or composer phpunit -- /path/to/tests/phpunit/unit.

Generating code coverage is now faster

The PHPUnit configuration file now resides at the root of the repository, and is called phpunit.xml.dist. (As an aside, you can copy this to phpunit.xml and make local changes, as that file is git-ignored, although you should not need to do that.) We made a modification (T192078) to the PHPUnit configuration inside MediaWiki to speed up code coverage generation. This makes it feasible to have a split window in your IDE (e.g. PhpStorm), run "Debug with coverage", and see the results in your editor fairly quickly after running the tests.

Debug coverage in PhpStorm

What is next?

Things we are working on:

  • Porting core tests to integration/unit
  • Porting extension tests to integration/unit.
  • Removing legacy testsuites or ensuring they can be run in a different way (passing the directory name for example).
  • Switching CI to use new entrypoint for unit tests, then for unit and integration tests

Help is wanted in all areas of the above! We can be found in the #wikimedia-codehealth channel and via the phab issues linked in this post.

Credits

The above work has been done and supported by Máté (@TK-999), Amir (@Ladsgroup), Kosta (@kostajh), James (@Jdforrester-WMF), Timo (@Krinkle), Leszek (@WMDE-leszek), Kunal (@Legoktm), Daniel (@daniel), Michael Große (@Michael), Adam (@awight), Antoine (@hashar), JR (@Jrbranaa) and Greg (@greg) along with several others. Thank you!

thanks for reading, and happy testing!

Amir, Kosta, & Máté

Written by kostajh on Jul 16 2019, 4:13 AM.
Senior Software Engineer (Growth)
Projects
Subscribers
Florian, D3r1ck01, RazeSoldier and 7 others
Tokens
"Stroopwafel" token, awarded by bd808."Party Time" token, awarded by Ladsgroup."Party Time" token, awarded by D3r1ck01."Mountain of Wealth" token, awarded by Daimona."Barnstar" token, awarded by Mholloway."Orange Medal" token, awarded by Krinkle.

Event Timeline

Emm, I am confused about the two types of tests, integration and unit. I don't really understand the difference between the two test types. This looks like the test directory structure (Unit and Feature) of the Laravel.

Emm, I am confused about the two types of tests, integration and unit. I don't really understand the difference between the two test types. This looks like the test directory structure (Unit and Feature) of the Laravel.

Does this help? https://en.wikipedia.org/wiki/Unit_testing and https://en.wikipedia.org/wiki/Integration_testing

Emm, I am confused about the two types of tests, integration and unit. I don't really understand the difference between the two test types. This looks like the test directory structure (Unit and Feature) of the Laravel.

Does this help? https://en.wikipedia.org/wiki/Unit_testing and https://en.wikipedia.org/wiki/Integration_testing

Could you give an example, in the current files, which is the unit test and which is the integration test? :)

Could you give an example, in the current files, which is the unit test and which is the integration test? :)

This is an integration test: https://github.com/wikimedia/mediawiki/blob/master/tests/phpunit/includes/api/ApiQueryWatchlistIntegrationTest.php

This is a unit test: https://github.com/wikimedia/mediawiki/blob/master/tests/phpunit/unit/includes/diff/DiffTest.php

Thank you, I think I understand.

Emm, I am confused about the two types of tests, integration and unit. I don't really understand the difference between the two test types. This looks like the test directory structure (Unit and Feature) of the Laravel.

I came across this earlier today: https://www.mediawiki.org/wiki/Requests_for_comment/Unit_testing, in particular this section has a helpful overview

Thanks to everyone involved in this! I finally can execute unit tests in my ide (integration tests are still throwing an unexpected error, however, this seems to be related to IntelliJ, as they run on the cli). This is absolutely awesome, as it will make developing tests and features much more easy! Kudos to everyone who helped getting this done! :)

I finally can execute unit tests in my ide (integration tests are still throwing an unexpected error, however, this seems to be related to IntelliJ, as they run on the cli)

@Florian if you press "Play" next to a test, double check that the output in the console shows the PHPUnit configuration file location as /path/to/mediawiki/phpunit.xml.dist and not /path/to/mediawiki/tests/phpunit/suite.xml. In my setup, it keeps reverting back to the latter (in a different MediaWiki directory too!) until I click "Edit configuration" and tell it to use correct one.

This is absolutely awesome, as it will make developing tests and features much more easy! Kudos to everyone who helped getting this done! :)

Thanks!

@kostajh Thanks for your answer! :) I know found my problem, it was not related to the phpunit setup at all (as I already thought, as the unit tests worked well). It was related to the fact, that I use docker to run php, as well as my database and the webserver. And I simply missed to tell IntelliJ that it needs to start the php container for running the tests on the same network as the database in order to be able to connect to it. Now, both, the unit and integration tests run perfectly fine in my extension!

Btw.: Sorry for not providing any further details in my first comment, it would probably be more clear that there was a database exception if I would've done this :]