Page MenuHomePhabricator
Create Task
Maniphest T50217

Speed up MediaWiki PHPUnit build by running integration tests in parallel
Open, LowPublic
Actions

Description

PHPUnit jobs have been merged into one job. A few years ago we had separate categories that ran concurrently. They were merged for unrelated reasons, but it did slow things down.

Look into software that would help running these in parallel within the job (separate threads/processes)

See also:
T60771: Jenkins: MediaWiki extensions phpunit test should also run mediawiki core tests

Details

Reference
bz48217

Related Objects

Event Timeline

bzimport raised the priority of this task from to Low.Nov 22 2014, 1:18 AM
bzimport added a project: Continuous-Integration-Infrastructure.
bzimport set Reference to bz48217.
bzimport added a subscriber: Unknown Object (MLST).
Krinkle created this task.May 7 2013, 4:52 PM
hashar added a comment.Oct 10 2013, 7:29 PM

Gist from Sebastian Bergmann, PHPUnit creator:

partition-testsuite.php

Script that (making way too many assumptions) generates an Apache Ant build script to run a PHPUnit test suite in parallel.

https://gist.github.com/sebastianbergmann/4405417

Then assemble the generated JUnit XML files using:

merge-phpunit-xml.php

Script that merges multiple PHPUnit XML logfiles (JUnit XML) into one PHPUnit XML logfile.

https://gist.github.com/sebastianbergmann/4405658

demon removed a subscriber: demon.Dec 16 2014, 6:05 PM
Krinkle renamed this task from Jenkins: Merge seperate mediawiki-core-phpunit-{group} jobs and use parallel-phpunit to Use parallel-phpunit in MediaWIki PHPUnit test jobs.Feb 16 2015, 3:18 AM
Krinkle set Security to None.
Krinkle removed a subscriber: Unknown Object (MLST).
Krinkle renamed this task from Use parallel-phpunit in MediaWIki PHPUnit test jobs to Use parallel-phpunit in MediaWiki PHPUnit test jobs.Mar 3 2015, 4:38 PM
Krinkle renamed this task from Use parallel-phpunit in MediaWiki PHPUnit test jobs to Speed up MediaWiki PHPUnit build by running tests in parallel.Apr 1 2015, 4:08 AM
Krinkle updated the task description. (Show Details)
Krinkle added a comment.Apr 1 2015, 4:41 AM

This might work for standalone projects without a database, but I imagine in the case of MediaWiki core the bootstrapping has significant overhead. As well as concurrency issues. Neither of the mentioned frameworks supports parallelising by group.

I think our librarisation effort will effectively parallelise our tests by being in separate independently versioned projects that don't need all tests to run at once.

See also T93556: Unit tests using database are slower than needed.

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 6 2015, 11:43 AM
Legoktm added a subscriber: Legoktm.Feb 15 2016, 9:11 AM

For MediaWiki core, all parsertests now run in a separate job from other phpunit tests, roughly cutting the time in half.

Krinkle closed this task as Declined.Jan 13 2017, 12:24 AM
Krinkle added subscribers: tstarling, daniel.

Thanks to the work of @tstarling and @daniel recently, our Parser and Database-related tests are now much faster. It's also become easier to run locally and run only certain subsets of the tests. Considering the full test suite is under 4-5 minutes for Jenkins, and other jobs take longer, I don't think it's worth the complexity to try and run tests in parallel.

kostajh reopened this task as Open.Jun 17 2019, 5:54 PM
kostajh added a subscriber: kostajh.

I'd like to revisit this as an experiment using fastest. In my local environment using MediaWiki-Docker-Dev, running the full test suite takes 49 minutes. With fastest, it takes 39 minutes:

root@68e6b761fef5:/var/www/mediawiki# find tests/phpunit extensions/**/tests/phpunit skins/**/tests/phpunit -name "*Test.php" | ./vendor/liuggio/fastest/fastest "tests/phpunit/phpunit.php {};"
find: ‘extensions/**/tests/phpunit’: No such file or directory
find: ‘skins/**/tests/phpunit’: No such file or directory
- 662 shuffled test classes into the queue.
- Will be consumed by 4 parallel Processes.




662/662 [============================] 100% 39 mins 6.0 MiB

     8 failures.
[4] tests/phpunit/tests/MediaWikiTestCaseSchema2Test.phpUsing PHP 7.2.14
PHPUnit 6.5.14 by Sebastian Bergmann and contributors.

E

Time: 2.62 seconds, Memory: 30.00MB

There was 1 error:

1) MediaWikiTestCaseSchema2Test::testMediaWikiTestCaseSchemaTestOrder
Error: Class 'MediaWikiTestCaseSchema1Test' not found

/var/www/mediawiki/tests/phpunit/tests/MediaWikiTestCaseSchema2Test.php:25
/var/www/mediawiki/tests/phpunit/MediaWikiTestCase.php:427
/var/www/mediawiki/maintenance/doMaintenance.php:99

ERRORS!
Tests: 1, Assertions: 0, Errors: 1.

[2] tests/phpunit/includes/specials/SpecialShortpagesTest.phpUsing PHP 7.2.14
PHPUnit 6.5.14 by Sebastian Bergmann and contributors.

F

Time: 2.52 seconds, Memory: 30.00MB

There was 1 failure:

1) SpecialShortpagesTest::testGetQueryInfoRespectsContentNS with data set #0 (array(0, 6), array(), array(0, 6))
=== Logs generated by test case
[caches] [info] LocalisationCache: using store LCStoreNull {"private":false}
[wfDebug] [debug] SpecialPage::getContext called and $mContext is null. Return RequestContext::getMain(); for sanity {"private":false}
[caches] [info] LocalisationCache: using store LCStoreNull {"private":false}
===
Failed asserting that two arrays are equal.
--- Expected
+++ Actual
@@ @@
 Array (
     0 => 0
-    1 => 6

/var/www/mediawiki/tests/phpunit/includes/specials/SpecialShortpagesTest.php:30
/var/www/mediawiki/tests/phpunit/MediaWikiTestCase.php:427
/var/www/mediawiki/maintenance/doMaintenance.php:99

FAILURES!
Tests: 1, Assertions: 3, Failures: 1.

[1] tests/phpunit/includes/search/SearchResultTest.php
Fatal error: Class 'MediawikiTestCase' not found in /var/www/mediawiki/tests/phpunit/includes/search/SearchResultTest.php on line 3

Call Stack:
    0.0087     406392   1. {main}() /var/www/mediawiki/tests/phpunit/phpunit.php:0
    0.0299     698800   2. require('/var/www/mediawiki/maintenance/doMaintenance.php') /var/www/mediawiki/tests/phpunit/phpunit.php:129
    0.3754   11037136   3. PHPUnitMaintClass->execute() /var/www/mediawiki/maintenance/doMaintenance.php:99
    0.4046   11902112   4. MediaWikiPHPUnitCommand->run(???, ???) /var/www/mediawiki/tests/phpunit/phpunit.php:90
    0.4508   12630272   5. MediaWikiTestRunner->getTest(???, ???, ???) /var/www/mediawiki/vendor/phpunit/phpunit/src/TextUI/Command.php:169
    0.4511   12630272   6. MediaWikiTestRunner->loadSuiteClass(???, ???) /var/www/mediawiki/vendor/phpunit/phpunit/src/Runner/BaseTestRunner.php:73
    0.4527   12647448   7. PHPUnit\Runner\StandardTestSuiteLoader->load(???, ???) /var/www/mediawiki/vendor/phpunit/phpunit/src/Runner/BaseTestRunner.php:130
    0.4529   12667984   8. PHPUnit\Util\Fileloader::checkAndLoad(???) /var/www/mediawiki/vendor/phpunit/phpunit/src/Runner/StandardTestSuiteLoader.php:43
    0.4534   12668240   9. PHPUnit\Util\Fileloader::load(???) /var/www/mediawiki/vendor/phpunit/phpunit/src/Util/Fileloader.php:48
    0.4541   12679072  10. include_once('/var/www/mediawiki/tests/phpunit/includes/search/SearchResultTest.php') /var/www/mediawiki/vendor/phpunit/phpunit/src/Util/Fileloader.php:64

Using PHP 7.2.14
PHP Fatal error:  Class 'MediawikiTestCase' not found in /var/www/mediawiki/tests/phpunit/includes/search/SearchResultTest.php on line 3
PHP Stack trace:
PHP   1. {main}() /var/www/mediawiki/tests/phpunit/phpunit.php:0
PHP   2. require() /var/www/mediawiki/tests/phpunit/phpunit.php:129
PHP   3. PHPUnitMaintClass->execute() /var/www/mediawiki/maintenance/doMaintenance.php:99
PHP   4. MediaWikiPHPUnitCommand->run($argv = *uninitialized*, $exit = *uninitialized*) /var/www/mediawiki/tests/phpunit/phpunit.php:90
PHP   5. MediaWikiTestRunner->getTest($suiteClassName = *uninitialized*, $suiteClassFile = *uninitialized*, $suffixes = *uninitialized*) /var/www/mediawiki/vendor/phpunit/phpunit/src/TextUI/Command.php:169
PHP   6. MediaWikiTestRunner->loadSuiteClass($suiteClassName = *uninitialized*, $suiteClassFile = *uninitialized*) /var/www/mediawiki/vendor/phpunit/phpunit/src/Runner/BaseTestRunner.php:73
PHP   7. PHPUnit\Runner\StandardTestSuiteLoader->load($suiteClassName = *uninitialized*, $suiteClassFile = *uninitialized*) /var/www/mediawiki/vendor/phpunit/phpunit/src/Runner/BaseTestRunner.php:130
PHP   8. PHPUnit\Util\Fileloader::checkAndLoad($filename = *uninitialized*) /var/www/mediawiki/vendor/phpunit/phpunit/src/Runner/StandardTestSuiteLoader.php:43
PHP   9. PHPUnit\Util\Fileloader::load($filename = *uninitialized*) /var/www/mediawiki/vendor/phpunit/phpunit/src/Util/Fileloader.php:48
PHP  10. include_once() /var/www/mediawiki/vendor/phpunit/phpunit/src/Util/Fileloader.php:64

[1] tests/phpunit/includes/libs/GenericArrayObjectTest.phpPHPUnit\Runner\Exception from line 102 of /var/www/mediawiki/vendor/phpunit/phpunit/src/Runner/StandardTestSuiteLoader.php: Class 'tests/phpunit/includes/libs/GenericArrayObjectTest' could not be found in '/var/www/mediawiki/tests/phpunit/includes/libs/GenericArrayObjectTest.php'.
#0 /var/www/mediawiki/vendor/phpunit/phpunit/src/Runner/BaseTestRunner.php(130): PHPUnit\Runner\StandardTestSuiteLoader->load('tests/phpunit/i...', '/var/www/mediaw...')
#1 /var/www/mediawiki/vendor/phpunit/phpunit/src/Runner/BaseTestRunner.php(73): PHPUnit\Runner\BaseTestRunner->loadSuiteClass('tests/phpunit/i...', '/var/www/mediaw...')
#2 /var/www/mediawiki/vendor/phpunit/phpunit/src/TextUI/Command.php(169): PHPUnit\Runner\BaseTestRunner->getTest('tests/phpunit/i...', '/var/www/mediaw...', Array)
#3 /var/www/mediawiki/tests/phpunit/phpunit.php(90): PHPUnit\TextUI\Command->run(Array, true)
#4 /var/www/mediawiki/maintenance/doMaintenance.php(99): PHPUnitMaintClass->execute()
#5 /var/www/mediawiki/tests/phpunit/phpunit.php(129): require('/var/www/mediaw...')
#6 {main}
Using PHP 7.2.14

[3] tests/phpunit/suites/ParserIntegrationTest.phpUsing PHP 7.2.14
[4ca2f19e580cc04e08f5ae71] [no req]   ArgumentCountError from line 44 of /var/www/mediawiki/tests/phpunit/suites/ParserIntegrationTest.php: Too few arguments to function ParserIntegrationTest::__construct(), 0 passed in /var/www/mediawiki/vendor/phpunit/phpunit/src/Framework/TestSuite.php on line 591 and exactly 3 expected
Backtrace:
#0 /var/www/mediawiki/vendor/phpunit/phpunit/src/Framework/TestSuite.php(591): ParserIntegrationTest->__construct()
#1 /var/www/mediawiki/vendor/phpunit/phpunit/src/Framework/TestSuite.php(882): PHPUnit\Framework\TestSuite::createTest(ReflectionClass, string)
#2 /var/www/mediawiki/vendor/phpunit/phpunit/src/Framework/TestSuite.php(187): PHPUnit\Framework\TestSuite->addTestMethod(ReflectionClass, ReflectionMethod)
#3 /var/www/mediawiki/vendor/phpunit/phpunit/src/Runner/BaseTestRunner.php(106): PHPUnit\Framework\TestSuite->__construct(ReflectionClass)
#4 /var/www/mediawiki/vendor/phpunit/phpunit/src/TextUI/Command.php(169): PHPUnit\Runner\BaseTestRunner->getTest(string, string, array)
#5 /var/www/mediawiki/tests/phpunit/phpunit.php(90): PHPUnit\TextUI\Command->run(array, boolean)
#6 /var/www/mediawiki/maintenance/doMaintenance.php(99): PHPUnitMaintClass->execute()
#7 /var/www/mediawiki/tests/phpunit/phpunit.php(129): require(string)
#8 {main}

[3] tests/phpunit/LessFileCompilationTest.phpUsing PHP 7.2.14
[b865182a94074440ab541361] [no req]   ArgumentCountError from line 28 of /var/www/mediawiki/tests/phpunit/LessFileCompilationTest.php: Too few arguments to function LessFileCompilationTest::__construct(), 0 passed in /var/www/mediawiki/vendor/phpunit/phpunit/src/Framework/TestSuite.php on line 591 and exactly 2 expected
Backtrace:
#0 /var/www/mediawiki/vendor/phpunit/phpunit/src/Framework/TestSuite.php(591): LessFileCompilationTest->__construct()
#1 /var/www/mediawiki/vendor/phpunit/phpunit/src/Framework/TestSuite.php(882): PHPUnit\Framework\TestSuite::createTest(ReflectionClass, string)
#2 /var/www/mediawiki/vendor/phpunit/phpunit/src/Framework/TestSuite.php(187): PHPUnit\Framework\TestSuite->addTestMethod(ReflectionClass, ReflectionMethod)
#3 /var/www/mediawiki/vendor/phpunit/phpunit/src/Runner/BaseTestRunner.php(106): PHPUnit\Framework\TestSuite->__construct(ReflectionClass)
#4 /var/www/mediawiki/vendor/phpunit/phpunit/src/TextUI/Command.php(169): PHPUnit\Runner\BaseTestRunner->getTest(string, string, array)
#5 /var/www/mediawiki/tests/phpunit/phpunit.php(90): PHPUnit\TextUI\Command->run(array, boolean)
#6 /var/www/mediawiki/maintenance/doMaintenance.php(99): PHPUnitMaintClass->execute()
#7 /var/www/mediawiki/tests/phpunit/phpunit.php(129): require(string)
#8 {main}

[4] tests/phpunit/docs/ExportDemoTest.php
Fatal error: Class 'DumpTestCase' not found in /var/www/mediawiki/tests/phpunit/docs/ExportDemoTest.php on line 11

Call Stack:
    0.0118     406360   1. {main}() /var/www/mediawiki/tests/phpunit/phpunit.php:0
    0.0490     698768   2. require('/var/www/mediawiki/maintenance/doMaintenance.php') /var/www/mediawiki/tests/phpunit/phpunit.php:129
    1.0348   11037104   3. PHPUnitMaintClass->execute() /var/www/mediawiki/maintenance/doMaintenance.php:99
    1.0749   11902080   4. MediaWikiPHPUnitCommand->run(???, ???) /var/www/mediawiki/tests/phpunit/phpunit.php:90
    1.2131   12630224   5. MediaWikiTestRunner->getTest(???, ???, ???) /var/www/mediawiki/vendor/phpunit/phpunit/src/TextUI/Command.php:169
    1.2143   12630224   6. MediaWikiTestRunner->loadSuiteClass(???, ???) /var/www/mediawiki/vendor/phpunit/phpunit/src/Runner/BaseTestRunner.php:73
    1.2188   12647400   7. PHPUnit\Runner\StandardTestSuiteLoader->load(???, ???) /var/www/mediawiki/vendor/phpunit/phpunit/src/Runner/BaseTestRunner.php:130
    1.2191   12667936   8. PHPUnit\Util\Fileloader::checkAndLoad(???) /var/www/mediawiki/vendor/phpunit/phpunit/src/Runner/StandardTestSuiteLoader.php:43
    1.2214   12668192   9. PHPUnit\Util\Fileloader::load(???) /var/www/mediawiki/vendor/phpunit/phpunit/src/Util/Fileloader.php:48
    1.2236   12672704  10. include_once('/var/www/mediawiki/tests/phpunit/docs/ExportDemoTest.php') /var/www/mediawiki/vendor/phpunit/phpunit/src/Util/Fileloader.php:64

Using PHP 7.2.14
PHP Fatal error:  Class 'DumpTestCase' not found in /var/www/mediawiki/tests/phpunit/docs/ExportDemoTest.php on line 11
PHP Stack trace:
PHP   1. {main}() /var/www/mediawiki/tests/phpunit/phpunit.php:0
PHP   2. require() /var/www/mediawiki/tests/phpunit/phpunit.php:129
PHP   3. PHPUnitMaintClass->execute() /var/www/mediawiki/maintenance/doMaintenance.php:99
PHP   4. MediaWikiPHPUnitCommand->run($argv = *uninitialized*, $exit = *uninitialized*) /var/www/mediawiki/tests/phpunit/phpunit.php:90
PHP   5. MediaWikiTestRunner->getTest($suiteClassName = *uninitialized*, $suiteClassFile = *uninitialized*, $suffixes = *uninitialized*) /var/www/mediawiki/vendor/phpunit/phpunit/src/TextUI/Command.php:169
PHP   6. MediaWikiTestRunner->loadSuiteClass($suiteClassName = *uninitialized*, $suiteClassFile = *uninitialized*) /var/www/mediawiki/vendor/phpunit/phpunit/src/Runner/BaseTestRunner.php:73
PHP   7. PHPUnit\Runner\StandardTestSuiteLoader->load($suiteClassName = *uninitialized*, $suiteClassFile = *uninitialized*) /var/www/mediawiki/vendor/phpunit/phpunit/src/Runner/BaseTestRunner.php:130
PHP   8. PHPUnit\Util\Fileloader::checkAndLoad($filename = *uninitialized*) /var/www/mediawiki/vendor/phpunit/phpunit/src/Runner/StandardTestSuiteLoader.php:43
PHP   9. PHPUnit\Util\Fileloader::load($filename = *uninitialized*) /var/www/mediawiki/vendor/phpunit/phpunit/src/Util/Fileloader.php:48
PHP  10. include_once() /var/www/mediawiki/vendor/phpunit/phpunit/src/Util/Fileloader.php:64

[4] tests/phpunit/includes/search/SearchEngineTest.phpUsing PHP 7.2.14
PHPUnit 6.5.14 by Sebastian Bergmann and contributors.

.........F

Time: 38.75 seconds, Memory: 34.00MB

There was 1 failure:

1) SearchEngineTest::testCompletionSearchMustRespectCapitalLinkOverrides with data set "Searching for "search is" will finds "search is not Search" on NS_CATEGORY" ('search is', 'Category:search is not Search', array(14))
=== Logs generated by test case
[caches] [info] LocalisationCache: using store LCStoreNull {"private":false}
[wfDebug] [debug] LocalisationCache::isExpired(en): cache missing, need to make one {"private":false}
[wfDebug] [debug] LocalisationCache::recache: got localisation for en from source {"private":false}
[caches] [info] LocalisationCache: using store LCStoreNull {"private":false}
===
Failed asserting that 0 matches expected 1.

/var/www/mediawiki/tests/phpunit/includes/search/SearchEngineTest.php:265
/var/www/mediawiki/tests/phpunit/MediaWikiTestCase.php:427
/var/www/mediawiki/maintenance/doMaintenance.php:99

FAILURES!
Tests: 10, Assertions: 34, Failures: 1.


    ✘ ehm broken tests...
    Time: 39 minutes 21 seconds 130 milliseconds, Memory: 6.00 MiB

In our CI environment where we have 8 vCPUs, this might go a lot faster.

There are some failures above due to expectation of running tests serially (e.g. testMediaWikiTestCaseSchemaTestOrder) but overall the test failures here don't seem to be insurmountable.

This might make more sense to pursue further after T90875: Convert tests/phpunit/phpunit.php entrypoint to plain PHPUnit with bootstrap file is done.

My rough plan would be:

  1. Patch quibble so that it accepts a --phpunit-command argument. (That will also be needed for T90875, so that you could run quibble --phpunt-command=vendor/bin/phpunit.)
  2. Submit a patch for integration/config with quibble --phpunit-command='find tests/phpunit extensions/**/tests/phpunit skins/**/tests/phpunit -name "*Test.php" | ./vendor/liuggio/fastest/fastest "tests/phpunit/phpunit.php {};"'
  3. Create a patch for core which adds https://github.com/liuggio/fastest to require-dev
  4. Pair with someone to do a test run in CI to check speed
kostajh renamed this task from Speed up MediaWiki PHPUnit build by running tests in parallel to Speed up MediaWiki PHPUnit build by running integration tests in parallel.Jun 17 2019, 5:55 PM
kostajh added subscribers: Ladsgroup, awight.
hashar added a comment.Jun 19 2019, 7:05 AM

There are a few gotchas though. Some testsuite define tests which would not be found by globbing files. Typically the parser tests. There are also alternative ways to register test files via the UnitTest hook.

running the full test suite takes 49 minutes. With fastest, it takes 39 minutes:

That suggest the suite is not that much CPU bound or surely running four in parallels would be way faster?

root@68e6b761fef5:/var/www/mediawiki# find tests/phpunit extensions/**/tests/phpunit skins/**/tests/phpunit -name "*Test.php" | ./vendor/liuggio/fastest/fastest "tests/phpunit/phpunit.php {};"
find: ‘extensions/**/tests/phpunit’: No such file or directory
find: ‘skins/**/tests/phpunit’: No such file or directory

From the command line, that seems to suggest to run test classes from mediawiki/core as well as all extensions and skins. Then it does not find any file for extensions or skins. So is that solely for mediawiki/core tests? And if so why does it take 49 minutes when being run serially? It should be ten time faster (3 mins 15 s on my computer with hhvm).

In our CI environment where we have 8 vCPUs, this might go a lot faster.

The devil there is that we do not have the capacity to add 8 times more CPU on the WMCS infra. Even if instances have 8 vCPUS, they currently can run up to 4 jobs in parallel and we still need lot of CPU cycles for eg MySQL :]

Krinkle moved this task from Inbox to PHPUnit on the MediaWiki-Core-Testing board.Sep 4 2019, 5:49 PM
Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL