PHPUnit 10 was released on February 3, 2023. According to the announcement, this release brings lots of changes to the internals of PHPUnit, like a new event system that replaces test listeners and hooks. It would also be interesting to see if upgrading would speed up our test suites. Note that PHPUnit 10 requires PHP >= 8.1, so we can't upgrade until prod is on 8.1. (T319432) and the minimum version requirement is bumped in the code. Also, we will need to make some changes to both the tests (for methods that were deprecated/removed) and the configuration (e.g., update the listeners we're using to the new event system). This task will be similar to T243600.
Incomplete list of known blockers/TODOs
- Everything covered by the 9.6 upgrade task T342110
- PHP 8.1+ dependency (T328921)
- Deprecation notices are emitted for non-static data providers (T332865)
- TestCase::run became final, currently overridden in MediaWikiIntegrationTestCase (T342259)
- Return type hint of mixed added to TestCase::runTest, currently overridden in MediaWikiUnitTestCase and ApiTestCase, function also now private
- Test::getLinesToBeCovered, used in MediaWikiCoversValidator, has been moved to PHPUnit\Metadata\Api\CodeCoverage::linesToBeCovered
- PHPUnit prints the PHP version before starting tests, so we can remove the code which does the same thing in our bootstrap
- johnkary/phpunit-speedtrap is currently not compatible with PHPUnit 10. A PR for this has been open for 3 years now.
- T395743: Use of package ockcyp/covers-validator in libaries is not compatible with PHPUnit 10
- Config file schema has changed. Can be fixed quite easily by running PHPUnit with --migrate-configuration, but this needs manual review (unsupported options are just quietly removed)
- Make sure that PHPUnit is configured to display notices, deprecations, etc:
displayDetailsOnTestsThatTriggerDeprecations="true" displayDetailsOnTestsThatTriggerErrors="true" displayDetailsOnTestsThatTriggerNotices="true" displayDetailsOnTestsThatTriggerWarnings="true" displayDetailsOnPhpunitDeprecations="true"
- Support for custom printers has been removed. We currently use this for MediaWikiPHPUnitResultPrinter, which adds the "Logs generated by test case" thing.
- Make sure that data providers do not return complex object structures. PHPUnit will stringify them before running tests, potentially leading to massive slowdowns. This has been previously observed as PHPUnit hanging for ~40 seconds before actually starting tests. See T328919#9138704.
- The order in which @before and similar methods are called has been changed. For some reason, it looks like it now calls the @before method in a subclass before the @before method in a parent class. This is a huge issue because our base test classes (Integration and UnitTestCase) use them, and for instance, MediaWikiLangTestCase also defines a @before method. Those are written with the assumption that the parent @before is called first. This may or may not be intentional: the code that lists hook methods pays attention to the ordering by using array_unshift (see HookMethods.php). The method which generates a list of class methods also takes inheritance into account and sorts methods accordingly, see Reflection.php. This piece of code was changed to sort methods highest to lowest, then the change was reverted, and then the revert was also reverted. I can confirm that in PHPUnit 10.3.2 the methods are sorted highest to lowest, which is the opposite of what we want (as long as array_unshift is used). But the code in HookMethods.php has been using array_unshift for a long time, before those changes to Reflection were made; additionally, the commit message of the Reflection change states "so that tests implemented in a parent class are run before the tests implemented in a child class", which doesn't mention hook methods. To me, it looks like Sebastian had no intention of changing the ordering of test methods, and simply forgot to update the array_unshift calls when reversing the ordering in Reflection. Note, this is also true, but in reverse, for @after and the other hook methods that are run after tests. Upstream issue: https://github.com/sebastianbergmann/phpunit/issues/5498 --> fixed in PHPUnit 10.3.3
- TestSuite is now officially impossible to extend; the static suite() method has been removed, and the constructor has been made final and private. It was already pretty clear that this was going to happen when we upgraded to PHPUnit 8 and had to introduce SuiteEventsTrait. We currently use custom suite classes for two main use cases: parser tests, skin tests, and extension tests (including unit tests after r937559). These will need to be re-done differently. Note that these custom test suites are also currently preventing us from using paratest (T50217), because the list of tests is not known soon enough. See T345481.
- Our custom PHPUnit extensions (currently MediaWikiLoggerPHPUnitExtension and MediaWikiTeardownPHPUnitExtension) must be updated with PHPUnit 10's event system.
- TestCase::$backupGlobalsExcludeList and TestCase::$backupStaticPropertiesExcludeList now private, remove set in MediaWikiIntegrationTestCase::__construct (ref)
- More stuff from T328919#10855098