PHPUnit 10 [[https://phpunit.de/announcements/phpunit-10.html | was released ]] a few days ago. According to the announcement, this release brings lots of changes to the internals of PHPUnit, like a new event system that replaces test listeners and hooks. It would also be interesting to see if upgrading would speed up our test suites. Note that PHPUnit 10 requires PHP >= 8.1, so we can't upgrade until prod is on 8.1. (T319432) and the minimum version requirement is bumped in the code. Also, we will need to make some changes to both the tests (for methods that were deprecated/removed) and the configuration (e.g., update the listeners we're using to the new event system). This task will be similar to T243600.
===Incomplete list of known blockers/TODOs===
[] Everything covered by the 9.6 upgrade task T342110
[] PHP 8.1+ dependency (T328922)
[] Deprecation notices are emitted for non-static data providers (T332865)
[] TestCase::run became final, currently overridden in MediaWikiIntegrationTestCase (T342259)
[] Return type hint of `mixed` added to TestCase::runTest, currently overridden in [[https://codesearch.wmcloud.org/search/?q=function+runTest%5C%28&files=phpunit%2F&excludeFiles=&repos= | MediaWikiUnitTestCase and ApiTestCase ]]
[] Test::getLinesToBeCovered, used in MediaWikiCoversValidator, has been moved to \PHPUnit\Metadata\Api::linesToBeCovered
[] PHPUnit prints the PHP version before starting tests, so we can remove the code which does the same thing in our bootstrap
[] [[https://github.com/johnkary/phpunit-speedtrap | johnkary/phpunit-speedtrap]] is currently not compatible with PHPUnit 10. A [[https://github.com/johnkary/phpunit-speedtrap/pull/98 | PR ]] for this has been open for 10 months now.
[] Config file schema has changed. Can be fixed quite easily by running PHPUnit with `--migrate-configuration`
[] Support for custom printers has been removed. We currently use this for MediaWikiPHPUnitResultPrinter, which adds the "Logs generated by test case" thing.
[] To figure out: it looks like the data provider phase runs MUCH slower in PHPUnit 10; locally, PHPUnit hangs for ~40 seconds before actually starting tests, as opposed to just a few seconds in PHPUnit 9. I still don't know what causes this, whether it's caused by the data provider changes, whether it's caused by our bootstrap code, etc.
[] The order in which `@before` and similar methods are called has been changed. For some reason, it looks like it now calls the `@before` method in a subclass before the `@before` method in a parent class. This is a huge issue because our base test classes (Integration and UnitTestCase) use them, and for instance, MediaWikiLangTestCase also defines a `@before` method. Those are written with the assumption that the parent `@before` is called first. This may or may not be intentional: the code that lists hook methods pays attention to the ordering by using array_unshift (see [[https://github.com/sebastianbergmann/phpunit/blob/7ef88cc9b9359dc771b80c61554232b27fd70228/src/Metadata/Api/HookMethods.php#L55 | HookMethods.php]]). The method which generates a list of class methods also takes inheritance into account and sorts methods accordingly, see [[https://github.com/sebastianbergmann/phpunit/blob/7ef88cc9b9359dc771b80c61554232b27fd70228/src/Util/Reflection.php#L69 | Reflection.php]]. This piece of code was [[https://github.com/sebastianbergmann/phpunit/commit/c748e1e850b9acba1db553d7eb64eaf48f181c5b | changed ]] to sort methods highest to lowest, then the change was [[https://github.com/sebastianbergmann/phpunit/commit/3d3c5be35795083909598f60615abc4345fe21c1 | reverted]], and then the revert was also [[https://github.com/sebastianbergmann/phpunit/commit/cbe7931144658511e82e88632894f3516f07594c | reverted ]]. I can confirm that in PHPUnit 10.3.2 the methods are sorted highest to lowest, which is the opposite of what we want (as long as array_unshift is used). But the code in HookMethods.php has been using array_unshift for a long time, before those changes to Reflection were made; additionally, the commit message of the Reflection change states "so that tests implemented in a parent class are run before the tests implemented in a child class", which doesn't mention hook methods. To me, it looks like Sebastian had no intention of changing the ordering of test methods, and simply forgot to update the array_unshift calls when reversing the ordering in Reflection. Note, this is also true, but in reverse, for `@after` and the other hook methods that are run after tests. Upstream issue: https://github.com/sebastianbergmann/phpunit/issues/5498
[] TestSuite is now officially impossible to extend; the static suite() method has been removed, and the constructor has been made final and private. It was already pretty clear that this was going to happen when we upgraded to PHPUnit 8 and had to introduce [[https://gerrit.wikimedia.org/g/mediawiki/core/+/965f722117315082d53e0fff53b4532bbc639121/tests/phpunit/suites/SuiteEventsTrait.php | SuiteEventsTrait]]. We currently use custom suite classes for [[https://codesearch.wmcloud.org/search/?q=extends+.*TestSuite%5Cb&files=phpunit%2F&excludeFiles=&repos= | two main use cases ]]: parser tests, and extension tests (including unit tests after [[https://gerrit.wikimedia.org/r/c/mediawiki/core/+/937559 | r937559]]). These will need to be re-done differently. Note that these custom test suites are also currently preventing us from using paratest (T50217), because the list of tests is not known soon enough. See T345481.
[] Our custom PHPUnit extensions (currently `MediaWikiLoggerPHPUnitExtension` and `MediaWikiTeardownPHPUnitExtension`; after [[https://gerrit.wikimedia.org/r/c/mediawiki/core/+/937980 | r937980]], `MediaWikiDeprecatedConfigPHPUnitExtension` as well) must be updated with PHPUnit 10's event system.