Background
Long ago, our Jenkins configuration used to include multiple jobs that run PHPUnit on MediaWiki core commits. Each job ran a portion of the tests (by passing a filter or subdirectory to the phpunit command), thus allowing them to run concurrently. Over the years, as things have been optimised and improved, these were eventually merged into one large job that basically "just" runs phpunit. This is how it has been for about 7 years now.
As per T225730: Reduce runtime of MW shared gate Jenkins jobs to 5 min, in the last 5 years the job has gotten significantly slower due to growth in how many tests we run, and so now the time developers spend waiting for a CI response to a MediaWiki commit, is spent in the PHPUnit job, and so we want to make it faster again. We believe the job is not exhausting its allocated resources and could run much faster, if it was parallelised somehow. By approaching it as a single job with concurrency (instead of splitting the jobs) we have two other benefits: 1) Keeps CI configuration simple, 2) Means it will also become fast by default for developers running tests locally.
Work
Look into software that would help running these in parallel within the job (separate threads/processes)
- https://github.com/brianium/paratest (promoted by https://phpunit.de/plugins.html)
- https://github.com/facile-it/paraunit
- https://github.com/verkkokauppacom/parallel-phpunit
See also:
T60771: Jenkins: MediaWiki extensions phpunit test should also run mediawiki core tests
Update in 2022 by @aaron:
- The most promising so far is paratest.
- A notable restriction is that it does not support a wrapped phpunit command. MediaWiki currently wraps phpunit in tests/phpunit.php, being fixed as part of T90875 is a blocker unless we use paratest 1.x (no longer maintained), which did support custom wrappers.
- Distribution of tests between subproceses should be invisible to developers in practice. There are multiple ways to do it, e.g.
- by "suite" (per top-level subdir of tests/phpunit/includes),
- by class (one Test.php file),
- by function (individual test cases, including such that e.g. the "setUpBeforeClass" hooks will run multiple times, once in each process that end sup running one or more of the test functions, and even individual data provided cases could end up split and e.g. clash or do more work than is needed). The "by function" is known to have many failures with our current setup and would take much effort to make pass and keep passing.
2025-08 status update
This task is not ready to be worked on, because we first need to address the following:
- Stop using "dynamic" PHPUnit suites (T345481, T358394). This is a prerequisite for both this task and the PHPUnit 10 upgrade (T328919), and it's a decent amount of work.
- Make the parallel setup work in SQLite (also for MediaWiki-Docker). Could probably reuse something from Aaron's config
- Update PHPUnit plugins so that they buffer their output from all processes and print it all at once at the end. Otherwise, paratest will just blackhole the output (see issue 771). This should be a relatively straightforward change but must be done upstream. We will need this at least for the "slow test detector" plugin, but note that we might switch to a different plugin because the one we're currently using is not compatible with PHPUnit 10.
- The ob_start in bootstrap.common.php used to cause issues with paratest, although I don't remember what issues exactly, and I don't know whether that is still the case.
Once the above are done, switching to paratest should be relatively straightforward.
