Building off the work done at the Prague Hackathon (T216260), we're happy to announce some significant changes and improvements to the PHP testing tools included with MediaWiki.
PHP unit tests can now be run statically, without installing MediaWiki
You can now download MediaWiki, run composer install, and then composer phpunit:unit to run core's unit test suite (T89432).
The standard PHPUnit entrypoint can be used, instead of the PHPUnit Maintenance class
You can now use the plain PHPUnit entrypoint at vendor/bin/phpunit instead of the MediaWiki maintenance class which wraps PHPUnit (tests/phpunit/phpunit.php).
Both the unit tests and integration tests can be executed with the standard phpunit entrypoint (vendor/bin/phpunit) or if you prefer, with the composer scripts defined in composer.json (e.g. composer phpunit:unit). We accomplished this by writing a new bootstrap.php file (the old one which the maintenance class uses was moved to tests/phpunit/bootstrap.maintenance.php) which executes the minimal amount of code necessary to make core, extension and skin classes discoverable by test classes.
Tests should be placed in tests/phpunit/{integration,unit}
Integration tests should be placed in tests/phpunit/integration while unit tests go in tests/phpunit/unit, these are discoverable by the new test suites (T87781). It sounds obvious now to write this, but a nice side effect is that by organizing tests into these directories it's immediately clear to authors and reviewers what type of test one is looking at.
Introducing MediaWikiUnitTestCase
A new base test case, MediaWikiUnitTestCase has been introduced with a minimal amount of boilerplate (@covers validator, ensuring the globals are disabled, and that the tests are in the proper directory, the default PHPUnit 4 and 6 compatibility layer). The MediaWikiTestCase has been renamed to MediaWikiIntegrationTestCase for clarity.
Please migrate tests to be unit tests where appropriate
A significant portion of core's unit tests have been ported to use MediaWikiUnitTestCase, approximately 50% of the total. We have also worked on porting extension tests to the unit/integration directories. @Ladsgroup wrote a helpful script to assist with automating the identification and moving of unit tests, see P8702. Migrating tests from MediaWikiIntegrationTestCase to MediaWikiUnitTestCase makes them faster.
Note that unit tests in CI are still run with the PHPUnit maintenance class (tests/phpunit/phpunit.php), so when reviewing unit test patches please execute them locally with vendor/bin/phpunit /path/to/tests/phpunit/unit or composer phpunit -- /path/to/tests/phpunit/unit.
Generating code coverage is now faster
The PHPUnit configuration file now resides at the root of the repository, and is called phpunit.xml.dist. (As an aside, you can copy this to phpunit.xml and make local changes, as that file is git-ignored, although you should not need to do that.) We made a modification (T192078) to the PHPUnit configuration inside MediaWiki to speed up code coverage generation. This makes it feasible to have a split window in your IDE (e.g. PhpStorm), run "Debug with coverage", and see the results in your editor fairly quickly after running the tests.
What is next?
Things we are working on:
- Porting core tests to integration/unit
- Porting extension tests to integration/unit.
- Removing legacy testsuites or ensuring they can be run in a different way (passing the directory name for example).
- Switching CI to use new entrypoint for unit tests, then for unit and integration tests
Help is wanted in all areas of the above! We can be found in the #wikimedia-codehealth channel and via the phab issues linked in this post.
Credits
The above work has been done and supported by Máté (@TK-999), Amir (@Ladsgroup), Kosta (@kostajh), James (@Jdforrester-WMF), Timo (@Krinkle), Leszek (@WMDE-leszek), Kunal (@Legoktm), Daniel (@daniel), Michael Große (@Michael), Adam (@awight), Antoine (@hashar), JR (@Jrbranaa) and Greg (@greg) along with several others. Thank you!
thanks for reading, and happy testing!
Amir, Kosta, & Máté
Event Timeline
Emm, I am confused about the two types of tests, integration and unit. I don't really understand the difference between the two test types. This looks like the test directory structure (Unit and Feature) of the Laravel.
Could you give an example, in the current files, which is the unit test and which is the integration test? :)
Emm, I am confused about the two types of tests, integration and unit. I don't really understand the difference between the two test types. This looks like the test directory structure (Unit and Feature) of the Laravel.
I came across this earlier today: https://www.mediawiki.org/wiki/Requests_for_comment/Unit_testing, in particular this section has a helpful overview
Thanks to everyone involved in this! I finally can execute unit tests in my ide (integration tests are still throwing an unexpected error, however, this seems to be related to IntelliJ, as they run on the cli). This is absolutely awesome, as it will make developing tests and features much more easy! Kudos to everyone who helped getting this done! :)
I finally can execute unit tests in my ide (integration tests are still throwing an unexpected error, however, this seems to be related to IntelliJ, as they run on the cli)
@Florian if you press "Play" next to a test, double check that the output in the console shows the PHPUnit configuration file location as /path/to/mediawiki/phpunit.xml.dist and not /path/to/mediawiki/tests/phpunit/suite.xml. In my setup, it keeps reverting back to the latter (in a different MediaWiki directory too!) until I click "Edit configuration" and tell it to use correct one.
This is absolutely awesome, as it will make developing tests and features much more easy! Kudos to everyone who helped getting this done! :)
Thanks!
@kostajh Thanks for your answer! :) I know found my problem, it was not related to the phpunit setup at all (as I already thought, as the unit tests worked well). It was related to the fact, that I use docker to run php, as well as my database and the webserver. And I simply missed to tell IntelliJ that it needs to start the php container for running the tests on the same network as the database in order to be able to connect to it. Now, both, the unit and integration tests run perfectly fine in my extension!
Btw.: Sorry for not providing any further details in my first comment, it would probably be more clear that there was a database exception if I would've done this :]
Are there any updates regarding the panned teck talk? https://www.youtube.com/watch?v=HOWKHUA-wAI It says 23. Nov, but there is no video available yet.
@Osnard it's been rescheduled for December 7 at 21.00 CEST, an email should go out next week.