Background
Long ago, our Jenkins configuration used to include multiple jobs that run PHPUnit on MediaWiki core commits. Each job ran a portion of the tests (by passing a filter or subdirectory to the phpunit command), thus allowing them to run concurrently. Over the years, as things have been optimised and improved, these were eventually merged into one large job that basically "just" runs phpunit. This is how it has been for about 7 years now.
As per T225730: Reduce runtime of MW shared gate Jenkins jobs to 5 min, in the last 5 years the job has gotten significantly slower due to growth in how many tests we run, and so now the time developers spend waiting for a CI response to a MediaWiki commit, is spent in the PHPUnit job, and so we want to make it faster again. We believe the job is not exhausting its allocated resources and could run much faster, if it was parallelised somehow. By approaching it as a single job with concurrency (instead of splitting the jobs) we have two other benefits: 1) Keeps CI configuration simple, 2) Means it will also become fast by default for developers running tests locally.
Work
Look into software that would help running these in parallel within the job (separate threads/processes)
- https://github.com/brianium/paratest (promoted by https://phpunit.de/plugins.html)
- https://github.com/facile-it/paraunit
- https://github.com/verkkokauppacom/parallel-phpunit
See also:
T60771: Jenkins: MediaWiki extensions phpunit test should also run mediawiki core tests
Update in 2022 by @aaron:
- The most promising so far is paratest.
- A notable restriction is that it does not support a wrapped phpunit command. MediaWiki currently wraps phpunit in tests/phpunit.php, being fixed as part of T90875 is a blocker unless we use paratest 1.x (no longer maintained), which did support custom wrappers.
- Distribution of tests between subproceses should be invisible to developers in practice. There are multiple ways to do it, e.g.
- by "suite" (per top-level subdir of tests/phpunit/includes),
- by class (one Test.php file),
- by function (individual test cases, including such that e.g. the "setUpBeforeClass" hooks will run multiple times, once in each process that end sup running one or more of the test functions, and even individual data provided cases could end up split and e.g. clash or do more work than is needed). The "by function" is known to have many failures with our current setup and would take much effort to make pass and keep passing.
