Page MenuHomePhabricator

Phase out usage of tests/phpunit/suite.xml
Open, MediumPublic

Description

We have phpunit.xml.dist in the root of the MediaWiki repo, and it's intended to use this file with vendor/bin/phpunit (or composer phpunit) rather than the PHPUnit MediaWiki maintenance class at tests/phpunit/phpunit.php.

In order to phase out the usage of tests/phpunit/suite.xml, we need to ensure that all tests currently run via the maintenance class can be run with the plain phpunit entrypoint. We also need to ensure that the test files are located in directories expected by the test suites defined in phpunit.xml.dist (e.g. unit and integration directories).

Event Timeline

Probably a starting point here is to run composer phpunit:integration in CI, maybe just starting with core rather than core + extensions. While it's pretty limited in scope now eventually this would lead to double the amount of execution time (since we're running the same tests twice). I don't have ideas for how else we'd manage this transition though. (I guess we could also have a discussion about whether we want this transition -- I would be in favor of it but if others are opposed, let's discuss.)

When trying out your composer phpunit locally, I run into a few issues that I don't encounter with tests/php/phpunit.php. It would be nice if these could be fixed somehow.

  1. It wants to run tests for all extensions and skins in the filesystem, not just the ones that are actually enabled on the local wiki.
    • I actually have my LocalSettings.php set up to not enable extensions by default when running PHPUnit tests. Just core's tests already take a long time to run.
  2. The convertWarningsToExceptions="true" and convertNoticesToExceptions="true" directives in phpunit.xml.dist seem to not be working. Possibly that's what phpunit.php is fixing with lines 50–54.
  3. How do I specify --wiki=foowiki? Integration tests, at least, can need this.
  4. With phpunit.php I could just do php7.4 tests/php/phpunit.php to run with a different version of PHP. How do I do that with composer?
  5. phpunit.php prints "Using PHP $VERSION" before actually running the tests, and locally I have some code in my LocalSettings.php to remind me of some other aspects of my environment are in effect for the test run (e.g. which wiki, and whether extensions are enabled). It would be nice to keep that. Currently LocalSettings.php doesn't get loaded until after several thousand tests were already run.
  6. T246076: Wrong definition of MW_ENTRY_POINT when integration tests run.

With phpunit.php I could just do php7.4 tests/php/phpunit.php to run with a different version of PHP. How do I do that with composer?

Just to respond to this point quickly, you should be able to do php7.4 $(which composer) phpunit:unit.

Just to respond to this point quickly, you should be able to do php7.4 $(which composer) phpunit:unit.

It doesn't though, at some point it switches back to /usr/bin/php.

Even worse is when I want to run tests using phpdbg -qrr.