Page MenuHomePhabricator

Speed up MediaWiki PHPUnit build by running integration tests in parallel
Open, HighPublic

Description

Background

Long ago, our Jenkins configuration used to include multiple jobs that run PHPUnit on MediaWiki core commits. Each job ran a portion of the tests (by passing a filter or subdirectory to the phpunit command), thus allowing them to run concurrently. Over the years, as things have been optimised and improved, these were eventually merged into one large job that basically "just" runs phpunit. This is how it has been for about 7 years now.

As per T225730: Reduce runtime of MW shared gate Jenkins jobs to 5 min, in the last 5 years the job has gotten significantly slower due to growth in how many tests we run, and so now the time developers spend waiting for a CI response to a MediaWiki commit, is spent in the PHPUnit job, and so we want to make it faster again. We believe the job is not exhausting its allocated resources and could run much faster, if it was parallelised somehow. By approaching it as a single job with concurrency (instead of splitting the jobs) we have two other benefits: 1) Keeps CI configuration simple, 2) Means it will also become fast by default for developers running tests locally.

Work

Look into software that would help running these in parallel within the job (separate threads/processes)

See also:
T60771: Jenkins: MediaWiki extensions phpunit test should also run mediawiki core tests

Update in 2022 by @aaron:

  • The most promising so far is paratest.
  • A notable restriction is that it does not support a wrapped phpunit command. MediaWiki currently wraps phpunit in tests/phpunit.php, being fixed as part of T90875 is a blocker unless we use paratest 1.x (no longer maintained), which did support custom wrappers.
  • Distribution of tests between subproceses should be invisible to developers in practice. There are multiple ways to do it, e.g.
    • by "suite" (per top-level subdir of tests/phpunit/includes),
    • by class (one Test.php file),
    • by function (individual test cases, including such that e.g. the "setUpBeforeClass" hooks will run multiple times, once in each process that end sup running one or more of the test functions, and even individual data provided cases could end up split and e.g. clash or do more work than is needed). The "by function" is known to have many failures with our current setup and would take much effort to make pass and keep passing.

Related Objects

StatusSubtypeAssignedTask
OpenNone
Openpmiazga
Resolvedkostajh
Resolvedkostajh
ResolvedKrinkle
Resolvedkostajh
Resolvedkostajh
Resolvedkostajh
ResolvedDaimona
Resolvedkostajh
ResolvedDaimona
Resolveddaniel
ResolvedBUG REPORTkostajh
ResolvedDaimona
ResolvedDaimona
OpenNone
OpenNone
ResolvedArthurTaylor
OpenArthurTaylor
OpenNone

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
@aaron wrote at T269894

Something like the phpunit/paratest wrapper at https://github.com/AaronSchulz/mediawiki-developer-tools would be nice to have in MediaWiki core.

  • Merge mediawiki-developer-tools suite.xml optimizations (or an improved version) into core (these breaks up the tests into more suites that can be run in parallel)
  • Add support for cloning to new sqlite DB files in CloneDatabase; this will replace wfCloneSqliteSchemaForParatest() from mediawiki-developer-tools
  • Add paratest to composer.json
  • Merge ParatestWrapperSettings.php into core from mediawiki-developer-tools
  • Include a simplified version of mediawiki-phpunit from mediawiki-developer-tools in core (e.g. just for "dev_phpunit" and "dev_paratest" modes)

Change 742199 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[mediawiki/core@master] [WIP] phpunit: Support splitting extension suite

https://gerrit.wikimedia.org/r/742199

Change 742200 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[integration/quibble@master] [WIP] Run PHPUnit tests in parallel

https://gerrit.wikimedia.org/r/742200

Change 742199 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[mediawiki/core@master] [WIP] phpunit: Support splitting extension suite

https://gerrit.wikimedia.org/r/742199

Change 742200 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[integration/quibble@master] [WIP] Run PHPUnit tests in parallel

https://gerrit.wikimedia.org/r/742200

To explain a bit about what I'm trying here:

  • In the short term, I'm skeptical about adding paratest or other wrappers like https://github.com/liuggio/fastest until T90875: Use vendor/bin/phpunit instead of tests/phpunit/phpunit.php is done; it seems like adding more code to wrap around tests/phpunit/phpunit.php will make removing it in favor of vendor/bin/phpunit that much harder. Longer term, having paratest would be nice to faciliate local test execution.
  • we have some parallelism utility code in Quibble that seemed simple to adapt for PHPUnit. In fact, it is quite easy to adapt, but it turns out that a bunch of our tests don't pass when run out of order, on their own, or even when they are executed with --testsuite={testSuiteName} ==testsuite={someOtherTestSuiteName}. So as @Krinkle linked to above, I'm trying to fix tests and making subtasks as needed under T297078: PHPUnit tests should pass when run on their own.

Change 742199 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[mediawiki/core@master] [WIP] phpunit: Support splitting extension suite

https://gerrit.wikimedia.org/r/742199

Change 742200 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[integration/quibble@master] [WIP] Run PHPUnit tests in parallel

https://gerrit.wikimedia.org/r/742200

To explain a bit about what I'm trying here:

  • In the short term, I'm skeptical about adding paratest or other wrappers like https://github.com/liuggio/fastest until T90875: Use vendor/bin/phpunit instead of tests/phpunit/phpunit.php is done; it seems like adding more code to wrap around tests/phpunit/phpunit.php will make removing it in favor of vendor/bin/phpunit that much harder. Longer term, having paratest would be nice to faciliate local test execution.
  • we have some parallelism utility code in Quibble that seemed simple to adapt for PHPUnit. In fact, it is quite easy to adapt, but it turns out that a bunch of our tests don't pass when run out of order, on their own, or even when they are executed with --testsuite={testSuiteName} ==testsuite={someOtherTestSuiteName}. So as @Krinkle linked to above, I'm trying to fix tests and making subtasks as needed under T297078: PHPUnit tests should pass when run on their own.

The last ones to fix (which are also harder because I can't reproduce them breaking when running locally) from https://integration.wikimedia.org/ci/job/integration-quibble-fullrun/504/console:

21:31:19 Time: 1.64 minutes, Memory: 904.50 MB
21:31:19 
21:31:19 There was 1 error:
21:31:19 
21:31:19 1) MediaWikiServicesTest::testDefaultServiceInstantiation
21:31:19 Use of GrowthExperimentsTaskSuggester service was deprecated in GrowthExperiments 1.35. [Called from Wikimedia\Services\ServiceContainer::createService in /workspace/src/vendor/wikimedia/services/src/ServiceContainer.php at line 447]
21:31:19 
21:31:19 /workspace/src/includes/debug/MWDebug.php:377
21:31:19 /workspace/src/includes/debug/MWDebug.php:353
21:31:19 /workspace/src/includes/debug/MWDebug.php:232
21:31:19 /workspace/src/includes/GlobalFunctions.php:1005
21:31:19 /workspace/src/extensions/GrowthExperiments/ServiceWiring.php:589
21:31:19 /workspace/src/vendor/wikimedia/services/src/ServiceContainer.php:447
21:31:19 /workspace/src/vendor/wikimedia/services/src/ServiceContainer.php:416
21:31:19 /workspace/src/includes/MediaWikiServices.php:291
21:31:19 /workspace/src/tests/phpunit/includes/MediaWikiServicesTest.php:386
21:31:19 /workspace/src/tests/phpunit/MediaWikiIntegrationTestCase.php:456
21:31:19 /workspace/src/tests/phpunit/phpunit.php:141
21:31:19 /workspace/src/tests/phpunit/phpunit.php:207
21:31:19 === Logs generated by test case
21:31:19 [objectcache] [debug] MainWANObjectCache using store {class} {"class":"EmptyBagOStuff"}
21:31:19 [localisation] [debug] LocalisationCache using store LCStoreNull []
21:31:19 [objectcache] [debug] MainObjectStash using store {class} {"class":"HashBagOStuff"}
21:31:19 [GlobalTitleFail] [info] RequestContext::getTitle called with no title set. {"exception":{}}
21:31:19 [localisation] [debug] LocalisationCache::isExpired(en): cache missing, need to make one []
21:31:19 [GlobalTitleFail] [info] RequestContext::getTitle called with no title set. {"exception":{}}
21:31:19 [GlobalTitleFail] [info] RequestContext::getTitle called with no title set. {"exception":{}}
21:31:19 [wfDebug] [debug] ParserFactory: using default preprocessor {"private":false}
21:31:19 [ParserCache] [debug] Creating ParserCache instance for pcache []
21:31:19 [MessageCache] [debug] MessageCache using store {class} {"class":"HashBagOStuff"}
21:31:19 [objectcache] [debug] fetchOrRegenerate(wikidbA-unittest_:page:8:0e6409a4846e9aee0a73fd5dfb53dd5e9839c4c6): miss, new value computed []
21:31:19 ===
21:31:19 
21:31:19 --
21:31:19 
21:31:19 There were 12 failures:
21:31:19 
21:31:19 1) MediaWiki\Tests\Revision\RevisionRendererTest::testGetRenderedRevision_multi
21:31:19 slot header
21:31:19 Failed asserting that '<div class="mw-parser-output"><p><a href="/index.php?title=Kittens&amp;action=edit&amp;redlink=1" class="new" title="Kittens (page does not exist)">Kittens</a>\n
21:31:19 </p><h1 class="mw-slot-header"><mediainfoslotheader /></h1><p><a href="/index.php?title=Goats&amp;action=edit&amp;redlink=1" class="new" title="Goats (page does not exist)">Goats</a>\n
21:31:19 </p></div>' contains ">aux<".
21:31:19 
21:31:19 /workspace/src/tests/phpunit/includes/Revision/RevisionRendererTest.php:420
21:31:19 /workspace/src/tests/phpunit/MediaWikiIntegrationTestCase.php:456
21:31:19 /workspace/src/tests/phpunit/phpunit.php:141
21:31:19 /workspace/src/tests/phpunit/phpunit.php:207
21:31:19 === Logs generated by test case
21:31:19 [objectcache] [debug] MainWANObjectCache using store {class} {"class":"EmptyBagOStuff"}
21:31:19 [localisation] [debug] LocalisationCache using store LCStoreNull []
21:31:19 [objectcache] [debug] MainObjectStash using store {class} {"class":"HashBagOStuff"}
21:31:19 [ContentHandler] [info] Registered handler for wikitext: WikitextContentHandler []
21:31:19 [localisation] [debug] LocalisationCache::isExpired(en): cache missing, need to make one []
21:31:19 [wfDebug] [debug] ParserFactory: using default preprocessor {"private":false}
21:31:19 [Wikibase] [debug] {method}: setting {settingName} was given as a closure, resolve it to {logValue} {"method":"Wikibase\\Lib\\SettingsArray::getSetting","settingName":"thisWikiIsTheRepo","logValue":"true"}
21:31:19 [Wikibase] [debug] {method}: setting {settingName} was given as a closure, resolve it to {logValue} {"method":"Wikibase\\Lib\\SettingsArray::getSetting","settingName":"entitySources","logValue":"array (\n  'local' => \n  array (\n    'entityNamespaces' => \n    array (\n      'item' => 120,\n      'property' => 122,\n      'mediainfo' => '6\/mediainfo',\n    ),\n    'repoDatabase' => false,\n    'baseUri' => 'http:\/\/127.0.0.1:9412\/entity\/',\n    'rdfNodeNamespacePrefix' => 'wd',\n    'rdfPredicateNamespacePrefix' => '',\n    'interwikiPrefix' => '',\n  ),\n)"}
21:31:19 [Wikibase] [debug] {method}: setting {settingName} was given as a closure, resolve it to {logValue} {"method":"Wikibase\\Lib\\SettingsArray::getSetting","settingName":"entitySources","logValue":"array (\n  'local' => \n  array (\n    'repoDatabase' => false,\n    'entityNamespaces' => \n    array (\n      'item' => '120\/main',\n      'property' => '122\/main',\n      'mediainfo' => '6\/mediainfo',\n    ),\n    'baseUri' => 'http:\/\/127.0.0.1:9412\/entity\/',\n    'rdfNodeNamespacePrefix' => 'wd',\n    'rdfPredicateNamespacePrefix' => '',\n    'interwikiPrefix' => '',\n  ),\n)"}
21:31:19 [Wikibase] [debug] {method}: setting {settingName} was given as a closure, resolve it to {logValue} {"method":"Wikibase\\Lib\\SettingsArray::getSetting","settingName":"itemAndPropertySourceName","logValue":"'local'"}
21:31:19 [Wikibase] [debug] {method}: setting {settingName} was given as a closure, resolve it to {logValue} {"method":"Wikibase\\Lib\\SettingsArray::getSetting","settingName":"siteGlobalID","logValue":"'wikidbA'"}
21:31:19 [MessageCache] [debug] MessageCache using store {class} {"class":"HashBagOStuff"}
21:31:19 [Wikibase] [debug] {method}: setting {settingName} was given as a closure, resolve it to {logValue} {"method":"Wikibase\\Lib\\SettingsArray::getSetting","settingName":"siteGroup","logValue":"NULL"}
21:31:19 [DuplicateParse] [debug] MediaWiki\Parser\ParserObserver::notifyParse: Possibly redundant parse! {"page":"ns0:MediaWiki\\Tests\\Revision\\RevisionRendererTest","rev":null,"options-hash":"canonical","trace":"#0 \/workspace\/src\/includes\/content\/ContentHandler.php(1728): MediaWiki\\Parser\\ParserObserver->notifyParse(Object(Title), NULL, Object(ParserOptions), Object(ParserOutput))\n#1 \/workspace\/src\/includes\/content\/Renderer\/ContentRenderer.php(47): ContentHandler->getParserOutput(Object(WikitextContent), Object(MediaWiki\\Content\\Renderer\\ContentParseParams))\n#2 \/workspace\/src\/includes\/Revision\/RenderedRevision.php(271): MediaWiki\\Content\\Renderer\\ContentRenderer->getParserOutput(Object(WikitextContent), Object(MediaWiki\\Page\\PageIdentityValue), NULL, Object(ParserOptions), true)\n#3 \/workspace\/src\/includes\/Revision\/RenderedRevision.php(238): MediaWiki\\Revision\\RenderedRevision->getSlotParserOutputUncached(Object(WikitextContent), true)\n#4 \/workspace\/src\/includes\/Revision\/RevisionRenderer.php(236): MediaWiki\\Revision\\RenderedRevision->getSlotParserOutput('aux', Array)\n#5 \/workspace\/src\/includes\/Revision\/RevisionRenderer.php(158): MediaWiki\\Revision\\RevisionRenderer->combineSlotOutput(Object(MediaWiki\\Revision\\RenderedRevision), Array)\n#6 [internal function]: MediaWiki\\Revision\\RevisionRenderer->MediaWiki\\Revision\\{closure}(Object(MediaWiki\\Revision\\RenderedRevision), Array)\n#7 \/workspace\/src\/includes\/Revision\/RenderedRevision.php(200): call_user_func(Object(Closure), Object(MediaWiki\\Revision\\RenderedRevision), Array)\n#8 \/workspace\/src\/tests\/phpunit\/includes\/Revision\/RevisionRendererTest.php(406): MediaWiki\\Revision\\RenderedRevision->getRevisionParserOutput()\n#9 \/workspace\/src\/vendor\/phpunit\/phpunit\/src\/Framework\/TestCase.php(1472): MediaWiki\\Tests\\Revision\\RevisionRendererTest->testGetRenderedRevision_multi()\n#10 \/workspace\/src\/vendor\/phpunit\/phpunit\/src\/Framework\/TestCase.php(1092): PHPUnit\\Framework\\TestCase->runTest()\n#11 \/workspace\/src\/vendor\/phpunit\/phpunit\/src\/Framework\/TestResult.php(703): PHPUnit\\Framework\\TestCase->runBare()\n#12 \/workspace\/src\/vendor\/phpunit\/phpunit\/src\/Framework\/TestCase.php(820): PHPUnit\\Framework\\TestResult->run(Object(MediaWiki\\Tests\\Revision\\RevisionRendererTest))\n#13 \/workspace\/src\/tests\/phpunit\/MediaWikiIntegrationTestCase.php(456): PHPUnit\\Framework\\TestCase->run(Object(PHPUnit\\Framework\\TestResult))\n#14 \/workspace\/src\/vendor\/phpunit\/phpunit\/src\/Framework\/TestSuite.php(627): MediaWikiIntegrationTestCase->run(Object(PHPUnit\\Framework\\TestResult))\n#15 \/workspace\/src\/vendor\/phpunit\/phpunit\/src\/Framework\/TestSuite.php(627): PHPUnit\\Framework\\TestSuite->run(Object(PHPUnit\\Framework\\TestResult))\n#16 \/workspace\/src\/vendor\/phpunit\/phpunit\/src\/Framework\/TestSuite.php(627): PHPUnit\\Framework\\TestSuite->run(Object(PHPUnit\\Framework\\TestResult))\n#17 \/workspace\/src\/vendor\/phpunit\/phpunit\/src\/TextUI\/TestRunner.php(656): PHPUnit\\Framework\\TestSuite->run(Object(PHPUnit\\Framework\\TestResult))\n#18 \/workspace\/src\/vendor\/phpunit\/phpunit\/src\/TextUI\/Command.php(235): PHPUnit\\TextUI\\TestRunner->doRun(Object(PHPUnit\\Framework\\TestSuite), Array, Array, true)\n#19 \/workspace\/src\/tests\/phpunit\/phpunit.php(141): PHPUnit\\TextUI\\Command->run(Array, true)\n#20 \/workspace\/src\/tests\/phpunit\/phpunit.php(207): PHPUnitMaintClass->execute()\n#21 {main}","previous-trace":"#0 \/workspace\/src\/includes\/content\/ContentHandler.php(1728): MediaWiki\\Parser\\ParserObserver->notifyParse(Object(Title), NULL, Object(ParserOptions), Object(ParserOutput))\n#1 \/workspace\/src\/includes\/content\/Renderer\/ContentRenderer.php(47): ContentHandler->getParserOutput(Object(WikitextContent), Object(MediaWiki\\Content\\Renderer\\ContentParseParams))\n#2 \/workspace\/src\/includes\/Revision\/RenderedRevision.php(271): MediaWiki\\Content\\Renderer\\ContentRenderer->getParserOutput(Object(WikitextContent), Object(MediaWiki\\Page\\PageIdentityValue), NULL, Object(ParserOptions), true)\n#3 \/workspace\/src\/includes\/Revision\/RenderedRevision.php(238): MediaWiki\\Revision\\RenderedRevision->getSlotParserOutputUncached(Object(WikitextContent), true)\n#4 \/workspace\/src\/includes\/Revision\/RevisionRenderer.php(236): MediaWiki\\Revision\\RenderedRevision->getSlotParserOutput('main', Array)\n#5 \/workspace\/src\/includes\/Revision\/RevisionRenderer.php(158): MediaWiki\\Revision\\RevisionRenderer->combineSlotOutput(Object(MediaWiki\\Revision\\RenderedRevision), Array)\n#6 [internal function]: MediaWiki\\Revision\\RevisionRenderer->MediaWiki\\Revision\\{closure}(Object(MediaWiki\\Revision\\RenderedRevision), Array)\n#7 \/workspace\/src\/includes\/Revision\/RenderedRevision.php(200): call_user_func(Object(Closure), Object(MediaWiki\\Revision\\RenderedRevision), Array)\n#8 \/workspace\/src\/tests\/phpunit\/includes\/Revision\/RevisionRendererTest.php(406): MediaWiki\\Revision\\RenderedRevision->getRevisionParserOutput()\n#9 \/workspace\/src\/vendor\/phpunit\/phpunit\/src\/Framework\/TestCase.php(1472): MediaWiki\\Tests\\Revision\\RevisionRendererTest->testGetRenderedRevision_multi()\n#10 \/workspace\/src\/vendor\/phpunit\/phpunit\/src\/Framework\/TestCase.php(1092): PHPUnit\\Framework\\TestCase->runTest()\n#11 \/workspace\/src\/vendor\/phpunit\/phpunit\/src\/Framework\/TestResult.php(703): PHPUnit\\Framework\\TestCase->runBare()\n#12 \/workspace\/src\/vendor\/phpunit\/phpunit\/src\/Framework\/TestCase.php(820): PHPUnit\\Framework\\TestResult->run(Object(MediaWiki\\Tests\\Revision\\RevisionRendererTest))\n#13 \/workspace\/src\/tests\/phpunit\/MediaWikiIntegrationTestCase.php(456): PHPUnit\\Framework\\TestCase->run(Object(PHPUnit\\Framework\\TestResult))\n#14 \/workspace\/src\/vendor\/phpunit\/phpunit\/src\/Framework\/TestSuite.php(627): MediaWikiIntegrationTestCase->run(Object(PHPUnit\\Framework\\TestResult))\n#15 \/workspace\/src\/vendor\/phpunit\/phpunit\/src\/Framework\/TestSuite.php(627): PHPUnit\\Framework\\TestSuite->run(Object(PHPUnit\\Framework\\TestResult))\n#16 \/workspace\/src\/vendor\/phpunit\/phpunit\/src\/Framework\/TestSuite.php(627): PHPUnit\\Framework\\TestSuite->run(Object(PHPUnit\\Framework\\TestResult))\n#17 \/workspace\/src\/vendor\/phpunit\/phpunit\/src\/TextUI\/TestRunner.php(656): PHPUnit\\Framework\\TestSuite->run(Object(PHPUnit\\Framework\\TestResult))\n#18 \/workspace\/src\/vendor\/phpunit\/phpunit\/src\/TextUI\/Command.php(235): PHPUnit\\TextUI\\TestRunner->doRun(Object(PHPUnit\\Framework\\TestSuite), Array, Array, true)\n#19 \/workspace\/src\/tests\/phpunit\/phpunit.php(141): PHPUnit\\TextUI\\Command->run(Array, true)\n#20 \/workspace\/src\/tests\/phpunit\/phpunit.php(207): PHPUnitMaintClass->execute()\n#21 {main}"}
21:31:19 ===
21:31:19 
21:31:19 2) MediaWiki\Tests\Parser\ParserCacheTest::testJsonEncodeUnicode
21:31:19 Failed asserting that '' contains "Э".
21:31:19 
21:31:19 /workspace/src/tests/phpunit/includes/parser/ParserCacheTest.php:762
21:31:19 /workspace/src/tests/phpunit/MediaWikiIntegrationTestCase.php:456
21:31:19 /workspace/src/tests/phpunit/phpunit.php:141
21:31:19 /workspace/src/tests/phpunit/phpunit.php:207
21:31:19 === Logs generated by test case
21:31:19 [objectcache] [debug] MainWANObjectCache using store {class} {"class":"EmptyBagOStuff"}
21:31:19 [localisation] [debug] LocalisationCache using store LCStoreNull []
21:31:19 [objectcache] [debug] MainObjectStash using store {class} {"class":"HashBagOStuff"}
21:31:19 [ContentHandler] [info] Registered handler for wikitext: WikitextContentHandler []
21:31:19 ===
21:31:19 
21:31:19 3) NamespaceInfoTest::testGetCanonicalNamespaces
21:31:19 Failed asserting that two arrays are identical.
21:31:19 --- Expected
21:31:19 +++ Actual
21:31:19 @@ @@
21:31:19      3 => 'User_talk'
21:31:19      -1 => 'Special'
21:31:19      -2 => 'Media'
21:31:19 +    250 => 'Page'
21:31:19 +    251 => 'Page_talk'
21:31:19 +    252 => 'Index'
21:31:19 +    253 => 'Index_talk'
21:31:19 +    710 => 'TimedText'
21:31:19 +    711 => 'TimedText_talk'
21:31:19  )
21:31:19 
21:31:19 /workspace/src/tests/phpunit/includes/title/NamespaceInfoTest.php:842
21:31:19 /workspace/src/tests/phpunit/MediaWikiIntegrationTestCase.php:456
21:31:19 /workspace/src/tests/phpunit/phpunit.php:141
21:31:19 /workspace/src/tests/phpunit/phpunit.php:207
21:31:19 === Logs generated by test case
21:31:19 [objectcache] [debug] MainWANObjectCache using store {class} {"class":"EmptyBagOStuff"}
21:31:19 ===
21:31:19 
21:31:19 4) NamespaceInfoTest::testGetValidNamespaces
21:31:19 Failed asserting that two arrays are identical.
21:31:19 --- Expected
21:31:19 +++ Actual
21:31:19 @@ @@
21:31:19      1 => 1
21:31:19      2 => 2
21:31:19      3 => 3
21:31:19 +    4 => 250
21:31:19 +    5 => 251
21:31:19 +    6 => 252
21:31:19 +    7 => 253
21:31:19 +    8 => 710
21:31:19 +    9 => 711
21:31:19  )
21:31:19 
21:31:19 /workspace/src/tests/phpunit/includes/title/NamespaceInfoTest.php:896
21:31:19 /workspace/src/tests/phpunit/MediaWikiIntegrationTestCase.php:456
21:31:19 /workspace/src/tests/phpunit/phpunit.php:141
21:31:19 /workspace/src/tests/phpunit/phpunit.php:207
21:31:19 === Logs generated by test case
21:31:19 [objectcache] [debug] MainWANObjectCache using store {class} {"class":"EmptyBagOStuff"}
21:31:19 ===
21:31:19 
21:31:19 5) NamespaceInfoTest::testGetCanonicalNamespaces_NoCanonicalNamespaceNames
21:31:19 Failed asserting that two arrays are identical.
21:31:19 --- Expected
21:31:19 +++ Actual
21:31:19 @@ @@
21:31:19  Array &0 (
21:31:19      0 => ''
21:31:19 +    250 => 'Page'
21:31:19 +    251 => 'Page_talk'
21:31:19 +    252 => 'Index'
21:31:19 +    253 => 'Index_talk'
21:31:19 +    710 => 'TimedText'
21:31:19 +    711 => 'TimedText_talk'
21:31:19  )
21:31:19 
21:31:19 /workspace/src/tests/phpunit/includes/title/NamespaceInfoTest.php:911
21:31:19 /workspace/src/tests/phpunit/MediaWikiIntegrationTestCase.php:456
21:31:19 /workspace/src/tests/phpunit/phpunit.php:141
21:31:19 /workspace/src/tests/phpunit/phpunit.php:207
21:31:19 === Logs generated by test case
21:31:19 [objectcache] [debug] MainWANObjectCache using store {class} {"class":"EmptyBagOStuff"}
21:31:19 ===
21:31:19 
21:31:19 6) NamespaceInfoTest::testGetValidNamespaces_NoCanonicalNamespaceNames
21:31:19 Failed asserting that two arrays are identical.
21:31:19 --- Expected
21:31:19 +++ Actual
21:31:19 @@ @@
21:31:19  Array &0 (
21:31:19      0 => 0
21:31:19 +    1 => 250
21:31:19 +    2 => 251
21:31:19 +    3 => 252
21:31:19 +    4 => 253
21:31:19 +    5 => 710
21:31:19 +    6 => 711
21:31:19  )
21:31:19 
21:31:19 /workspace/src/tests/phpunit/includes/title/NamespaceInfoTest.php:940
21:31:19 /workspace/src/tests/phpunit/MediaWikiIntegrationTestCase.php:456
21:31:19 /workspace/src/tests/phpunit/phpunit.php:141
21:31:19 /workspace/src/tests/phpunit/phpunit.php:207
21:31:19 === Logs generated by test case
21:31:19 [objectcache] [debug] MainWANObjectCache using store {class} {"class":"EmptyBagOStuff"}
21:31:19 ===
21:31:19 
21:31:19 7) NamespaceInfoTest::testGetCanonicalNamespaces_ExtensionNamespaces
21:31:19 Failed asserting that two arrays are identical.
21:31:19 --- Expected
21:31:19 +++ Actual
21:31:19 @@ @@
21:31:19      -1 => 'Special'
21:31:19      -2 => 'Media'
21:31:19      12345 => 'Extended'
21:31:19 +    250 => 'Page'
21:31:19 +    251 => 'Page_talk'
21:31:19 +    252 => 'Index'
21:31:19 +    253 => 'Index_talk'
21:31:19 +    710 => 'TimedText'
21:31:19 +    711 => 'TimedText_talk'
21:31:19  )
21:31:19 
21:31:19 /workspace/src/tests/phpunit/includes/title/NamespaceInfoTest.php:963
21:31:19 /workspace/src/tests/phpunit/MediaWikiIntegrationTestCase.php:456
21:31:19 /workspace/src/tests/phpunit/phpunit.php:141
21:31:19 /workspace/src/tests/phpunit/phpunit.php:207
21:31:19 === Logs generated by test case
21:31:19 [objectcache] [debug] MainWANObjectCache using store {class} {"class":"EmptyBagOStuff"}
21:31:19 ===
21:31:19 
21:31:19 8) NamespaceInfoTest::testGetValidNamespaces_ExtensionNamespaces
21:31:19 Failed asserting that two arrays are identical.
21:31:19 --- Expected
21:31:19 +++ Actual
21:31:19 @@ @@
21:31:19      1 => 1
21:31:19      2 => 2
21:31:19      3 => 3
21:31:19 -    4 => 12345
21:31:19 +    4 => 250
21:31:19 +    5 => 251
21:31:19 +    6 => 252
21:31:19 +    7 => 253
21:31:19 +    8 => 710
21:31:19 +    9 => 711
21:31:19 +    10 => 12345
21:31:19  )
21:31:19 
21:31:19 /workspace/src/tests/phpunit/includes/title/NamespaceInfoTest.php:999
21:31:19 /workspace/src/tests/phpunit/MediaWikiIntegrationTestCase.php:456
21:31:19 /workspace/src/tests/phpunit/phpunit.php:141
21:31:19 /workspace/src/tests/phpunit/phpunit.php:207
21:31:19 === Logs generated by test case
21:31:19 [objectcache] [debug] MainWANObjectCache using store {class} {"class":"EmptyBagOStuff"}
21:31:19 ===
21:31:19 
21:31:19 9) NamespaceInfoTest::testGetCanonicalNamespaces_ExtraNamespaces
21:31:19 Failed asserting that two arrays are identical.
21:31:19 --- Expected
21:31:19 +++ Actual
21:31:19 @@ @@
21:31:19      -1 => 'Special'
21:31:19      -2 => 'Media'
21:31:19      1234567 => 'Extra'
21:31:19 +    250 => 'Page'
21:31:19 +    251 => 'Page_talk'
21:31:19 +    252 => 'Index'
21:31:19 +    253 => 'Index_talk'
21:31:19 +    710 => 'TimedText'
21:31:19 +    711 => 'TimedText_talk'
21:31:19  )
21:31:19 
21:31:19 /workspace/src/tests/phpunit/includes/title/NamespaceInfoTest.php:1089
21:31:19 /workspace/src/tests/phpunit/MediaWikiIntegrationTestCase.php:456
21:31:19 /workspace/src/tests/phpunit/phpunit.php:141
21:31:19 /workspace/src/tests/phpunit/phpunit.php:207
21:31:19 === Logs generated by test case
21:31:19 [objectcache] [debug] MainWANObjectCache using store {class} {"class":"EmptyBagOStuff"}
21:31:19 ===
21:31:19 
21:31:19 10) NamespaceInfoTest::testGetValidNamespaces_ExtraNamespaces
21:31:19 Failed asserting that two arrays are identical.
21:31:19 --- Expected
21:31:19 +++ Actual
21:31:19 @@ @@
21:31:19      1 => 1
21:31:19      2 => 2
21:31:19      3 => 3
21:31:19 -    4 => 1234567
21:31:19 +    4 => 250
21:31:19 +    5 => 251
21:31:19 +    6 => 252
21:31:19 +    7 => 253
21:31:19 +    8 => 710
21:31:19 +    9 => 711
21:31:19 +    10 => 1234567
21:31:19  )
21:31:19 
21:31:19 /workspace/src/tests/phpunit/includes/title/NamespaceInfoTest.php:1121
21:31:19 /workspace/src/tests/phpunit/MediaWikiIntegrationTestCase.php:456
21:31:19 /workspace/src/tests/phpunit/phpunit.php:141
21:31:19 /workspace/src/tests/phpunit/phpunit.php:207
21:31:19 === Logs generated by test case
21:31:19 [objectcache] [debug] MainWANObjectCache using store {class} {"class":"EmptyBagOStuff"}
21:31:19 ===
21:31:19 
21:31:19 11) NamespaceInfoTest::testGetCanonicalNamespaces_caching
21:31:19 Failed asserting that two arrays are identical.
21:31:19 --- Expected
21:31:19 +++ Actual
21:31:19 @@ @@
21:31:19      3 => 'User_talk'
21:31:19      -1 => 'Special'
21:31:19      -2 => 'Media'
21:31:19 +    250 => 'Page'
21:31:19 +    251 => 'Page_talk'
21:31:19 +    252 => 'Index'
21:31:19 +    253 => 'Index_talk'
21:31:19 +    710 => 'TimedText'
21:31:19 +    711 => 'TimedText_talk'
21:31:19  )
21:31:19 
21:31:19 /workspace/src/tests/phpunit/includes/title/NamespaceInfoTest.php:1144
21:31:19 /workspace/src/tests/phpunit/MediaWikiIntegrationTestCase.php:456
21:31:19 /workspace/src/tests/phpunit/phpunit.php:141
21:31:19 /workspace/src/tests/phpunit/phpunit.php:207
21:31:19 === Logs generated by test case
21:31:19 [objectcache] [debug] MainWANObjectCache using store {class} {"class":"EmptyBagOStuff"}
21:31:19 ===
21:31:19 
21:31:19 12) NamespaceInfoTest::testGetValidNamespaces_caching
21:31:19 Failed asserting that two arrays are identical.
21:31:19 --- Expected
21:31:19 +++ Actual
21:31:19 @@ @@
21:31:19      1 => 1
21:31:19      2 => 2
21:31:19      3 => 3
21:31:19 +    4 => 250
21:31:19 +    5 => 251
21:31:19 +    6 => 252
21:31:19 +    7 => 253
21:31:19 +    8 => 710
21:31:19 +    9 => 711
21:31:19  )
21:31:19 
21:31:19 /workspace/src/tests/phpunit/includes/title/NamespaceInfoTest.php:1203
21:31:19 /workspace/src/tests/phpunit/MediaWikiIntegrationTestCase.php:456
21:31:19 /workspace/src/tests/phpunit/phpunit.php:141
21:31:19 /workspace/src/tests/phpunit/phpunit.php:207
21:31:19 === Logs generated by test case
21:31:19 [objectcache] [debug] MainWANObjectCache using store {class} {"class":"EmptyBagOStuff"}
21:31:19 ===

Change 758784 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[mediawiki/core@master] [WIP] phpunit: Use process ID in test DB prefix

https://gerrit.wikimedia.org/r/758784

Change 581131 abandoned by Aaron Schulz:

[mediawiki/core@master] tests: split includes/ tests into suites for tools like paratest

Reason:

Going with automatic generation from paratest wrapper

https://gerrit.wikimedia.org/r/581131

Change 763650 had a related patch set uploaded (by Krinkle; author: Aaron Schulz):

[mediawiki/core@master] phpunit: Remove redundancy from \"skins\" test and \"extensions\" suites

https://gerrit.wikimedia.org/r/763650

Change 763650 abandoned by Aaron Schulz:

[mediawiki/core@master] phpunit: Remove redundancy from \"skins\" test and \"extensions\" suites

Reason:

Worked around this in paratest scripts

https://gerrit.wikimedia.org/r/763650

Change 763651 had a related patch set uploaded (by Krinkle; author: Aaron Schulz):

[mediawiki/core@master] tests: Avoid unsafe use of setUpBeforeClass() in ApiFormatXmlTest

https://gerrit.wikimedia.org/r/763651

Change 767232 had a related patch set uploaded (by Krinkle; author: Aaron Schulz):

[mediawiki/core@master] tests: Split out DatabaseSqliteUpgradeTest class

https://gerrit.wikimedia.org/r/767232

Change 763651 merged by jenkins-bot:

[mediawiki/core@master] tests: Avoid unsafe use of setUpBeforeClass() in ApiFormatXmlTest

https://gerrit.wikimedia.org/r/763651

Change 767232 merged by jenkins-bot:

[mediawiki/core@master] tests: Split out DatabaseSqliteUpgradeTest class

https://gerrit.wikimedia.org/r/767232

Change 776058 had a related patch set uploaded (by Krinkle; author: Aaron Schulz):

[mediawiki/core@master] phpunit: parse any --boostrap parameter in getopt() call

https://gerrit.wikimedia.org/r/776058

Change 776058 merged by jenkins-bot:

[mediawiki/core@master] phpunit: parse any --boostrap parameter in getopt() call

https://gerrit.wikimedia.org/r/776058

Change 793140 had a related patch set uploaded (by Krinkle; author: Aaron Schulz):

[mediawiki/core@master] tests: Fix memcached test failure with multiple BagOStuff test classes

https://gerrit.wikimedia.org/r/793140

Change 793140 merged by jenkins-bot:

[mediawiki/core@master] tests: Fix memcached test failure with multiple BagOStuff test classes

https://gerrit.wikimedia.org/r/793140

Now that T90875 is almost done, I think this should be unblocked. We still have some cleanup to do for T90875, but that's mostly documentation; and then there's T227900 for having a single config and bootstrap, but I don't know if that would affect paratest.

I've fiddled a bit with paratest locally. It's easier (albeit almost useless) with unit tests because our unit tests are pretty self-contained as they should be and do not need a database, so paratest works out of the box for them. composer phpunit:unit took 8.6 seconds, whereas vendor/bin/paratest --colors=always --testsuite=core:unit,extensions:unit,skins:unit --runner WrapperRunner took 3.5 seconds (with 16 processes).

I then wanted to try integration (non-database) tests. At the moment this doesn't work out of the box due to T155147 causing some DB errors, but I checked out r937980 and applied r938387 to skip the DB setup. This results in a lot of errors, but at least all the tests can run. vendor/bin/phpunit --exclude-group Broken,ParserFuzz,Stub,Database,Standalone took 3m36s, whereas vendor/bin/paratest --exclude-group Broken,ParserFuzz,Stub,Database,Standalone --runner WrapperRunner took 36 seconds. For some reason the paratest run executed 26k tests instead of 30k and I don't know why, but I'm going to assume (and hope) that the 4k tests that didn't run for some reason aren't just the most expensive ones. Also, when I restricted the test run to the "includes" suite, it went from 1m5s to 19s.

Note that the "slow tests" report and our extension which prints logs after failures don't work with paratest (https://github.com/paratestphp/paratest/issues/771), but we that can be addressed later, when our tests will really be ready for parallelization.

I think paratest is one of the most promising things we can try for T225730. By splitting the suites we may be able to improve the run time even further. I don't know if the end result will really be a 6x speedup, but we should really, really try and get this done.

Adding to the above, the "missing" tests are from the extensions and skins suites (particularly unit tests, but also integration tests, I think). I suspect it might be because of our hacky subclassing of TestSuite, which we should really get rid of some day. If I remove those 4 suites from phpunit.xml.dist (as well as the core:unit suite, which is executed with or without paratest but shouldn't really add much), both runs have 14231 tests, and the time for a full run goes from 2m59s to 35 seconds. Another thing I noticed is that paratest seems to hang for roughly 10 seconds at some point towards the end; however, I think it's not due to a slow test, but rather due to it putting together the output from all processes (just an ineducated guess!). And since there are ~1000 failures due to tests using the database when they shouldn't, there's definitely a lot of output to put together and print. I'm quite convinced that this is the case because as soon as it unfreezes, it prints all the results. But I don't have a quick way to test that, and the random hacks I made to the paratest ResultPrinter didn't fix this. But we'll see better once T155147 is done.

Daimona raised the priority of this task from Low to High.Jul 19 2023, 3:01 PM

Another thing I noticed is that paratest seems to hang for roughly 10 seconds at some point towards the end; however, I think it's not due to a slow test, but rather due to it putting together the output from all processes (just an ineducated guess!). And since there are ~1000 failures due to tests using the database when they shouldn't, there's definitely a lot of output to put together and print. I'm quite convinced that this is the case because as soon as it unfreezes, it prints all the results. But I don't have a quick way to test that, and the random hacks I made to the paratest ResultPrinter didn't fix this. But we'll see better once T155147 is done.

I was wrong. It is actually LanguageConverterFactoryTest which takes something like 11 seconds on its own with paratest. I think it might be simply because it's a large test, but I haven't looked closely. At any rate, I've pulled r938387, applied r939777 to cut down on DB usage, deleted LanguageConverterFactoryTest.php and re-ran the includes suite (7054 tests).

$ vendor/bin/phpunit -c tests/phpunit/suite.xml --testsuite includes --exclude-group Broken,ParserFuzz,Stub,Database,Standalone

38 seconds.

$ vendor/bin/paratest -c tests/phpunit/suite.xml --testsuite includes --exclude-group Broken,ParserFuzz,Stub,Database,Standalone --runner WrapperRunner

8.5 seconds.

New benchmark! This time with r938387 checked out, and all non-database tests passing when DB access is disabled. Once again, I've also disabled LanguageConverterFactoryTest (by marking it as skipped).

$ vendor/bin/phpunit -c tests/phpunit/suite.xml --testsuite includes --exclude-group Broken,ParserFuzz,Stub,Database,Standalone

Time: 00:39.145, Memory: 521.00 MB

OK, but incomplete, skipped, or risky tests!
Tests: 7579, Assertions: 14014, Skipped: 607.
$ vendor/bin/paratest -c tests/phpunit/suite.xml --testsuite includes --exclude-group Broken,ParserFuzz,Stub,Database,Standalone --runner WrapperRunner

Time: 00:07.781, Memory: 115.00 MB

OK, but incomplete, skipped, or risky tests! 
Tests: 7579, Assertions: 14014, Skipped: 607.

This seems to confirm the 5x speedup observed previously.

When adding LanguageConverterFactoryTest back, the results are respectively

Time: 00:53.573, Memory: 523.00 MB

OK, but incomplete, skipped, or risky tests!
Tests: 7579, Assertions: 15916, Skipped: 156.

and

Time: 00:16.575, Memory: 115.00 MB

OK, but incomplete, skipped, or risky tests! 
Tests: 7579, Assertions: 15916, Skipped: 156.

which kinda sucks, but we can think about it later on.

And it's time again for a new benchmark :) Thanks to T342418, language creation is now much faster and so is LanguageConverterFactoryTest. Therefore, for the full 'includes' suite as-is:

$ vendor/bin/phpunit -c tests/phpunit/suite.xml --testsuite includes --exclude-group Broken,ParserFuzz,Stub,Database,Standalone

Time: 00:39.939, Memory: 503.50 MB

OK, but incomplete, skipped, or risky tests!
Tests: 7579, Assertions: 15918, Skipped: 156.
$ vendor/bin/paratest -c tests/phpunit/suite.xml --testsuite includes --exclude-group Broken,ParserFuzz,Stub,Database,Standalone --runner WrapperRunner

Time: 00:07.844, Memory: 115.00 MB

OK, but incomplete, skipped, or risky tests! 
Tests: 7579, Assertions: 15918, Skipped: 156.

This looks really promising. I'm still planning to do some general PHPUnit cleanup before tackling this task (e.g., T342428 and T342301), and then there'll be some other issues to figure out, for instance the way of specifying extension tests, whose list is dynamically generated and doesn't work with paratest. But I'm quite confident overall.

This time not a benchmark but a FYI: do NOT try to run paratest with DB tests in your local. Not even as an experiment. I did it a few times, and it truncated a few tables in my real DB. I have no idea why, nor have I looked into Aaron's config for paratest on sqlite, but whatever the cause, it definitely connects to the real DB at some point.

Change 758784 abandoned by Kosta Harlan:

[mediawiki/core@master] phpunit: Use process ID in test DB prefix

Reason:

https://gerrit.wikimedia.org/r/758784

Change 742199 abandoned by Kosta Harlan:

[mediawiki/core@master] [WIP] phpunit: Support splitting core and extension suites

Reason:

https://gerrit.wikimedia.org/r/742199

Change 742200 abandoned by Kosta Harlan:

[integration/quibble@master] [WIP] Run PHPUnit tests in parallel

Reason:

https://gerrit.wikimedia.org/r/742200

This time not a benchmark but a FYI: do NOT try to run paratest with DB tests in your local. Not even as an experiment. I did it a few times, and it truncated a few tables in my real DB. I have no idea why, nor have I looked into Aaron's config for paratest on sqlite, but whatever the cause, it definitely connects to the real DB at some point.

I think this may be from tests which extend the base PHPUnit classes, instead of MediaWikiUnitTest or MediaWikiIntegrationTest.

This time not a benchmark but a FYI: do NOT try to run paratest with DB tests in your local. Not even as an experiment. I did it a few times, and it truncated a few tables in my real DB. I have no idea why, nor have I looked into Aaron's config for paratest on sqlite, but whatever the cause, it definitely connects to the real DB at some point.

I think this may be from tests which extend the base PHPUnit classes, instead of MediaWikiUnitTest or MediaWikiIntegrationTest.

I believe I was able to reproduce the bug with WikiPageDbTest, which does extend MediaWikiIntegrationTestCase. Further, I remember writing something about this bug somewhere, but I can't find it right now, so maybe I didn't. Anyway, I read somewhere that SQLite can sometimes "forget" about the DB prefix when the connection is reloaded, or something like that, esp. in case of concurrent connections. I think this might be what Aaron's code in ParatestSettings.php is for; but again, I'm not an SQLite expert.

TBH, the results for r1005792 look really promising. That patch is just a POC, and has numerous issues that need to be fixed (both in the config/bootstrap and in the tests themselves), but the speed seems consistent with my local experiments: the core-only database suite finished in 50 seconds, vs the 4m54s it took for the parent patch. Note that, amongst the issues to be fixed, paratest seems to be unable to report the actual number of tests it's running (notice how it starts from 36551 and ends at 10075); and it's not even clear if it's running all the tests or not (the parent patch has 12349 tests in the DB suite). Still, these issues can be resolved as usual, and so far I don't think we've found any showstoppers.

Change 1006168 had a related patch set uploaded (by Daimona Eaytoy; author: Daimona Eaytoy):

[mediawiki/core@master] phpunit: Do not crash when paratest options are passed in

https://gerrit.wikimedia.org/r/1006168

Note that, amongst the issues to be fixed, paratest seems to be unable to report the actual number of tests it's running (notice how it starts from 36551 and ends at 10075); and it's not even clear if it's running all the tests or not (the parent patch has 12349 tests in the DB suite).

That seems kind of worrying, though. If we switch the coverage job to use paratest as well, then we'd have some additional assurances that we're still covering the same code.

Note that, amongst the issues to be fixed, paratest seems to be unable to report the actual number of tests it's running (notice how it starts from 36551 and ends at 10075); and it's not even clear if it's running all the tests or not (the parent patch has 12349 tests in the DB suite).

That seems kind of worrying, though. If we switch the coverage job to use paratest as well, then we'd have some additional assurances that we're still covering the same code.

It is, and that's why I consider this a blocker. While I'm not 100% sure about it, I think this bug is a consequence of "dynamic" suites (T345481), as I'm unable to reproduce it otherwise. My proposed fix for extension tests is already applied in the test run above, but we still need to fix parser tests and others, most notably scribunto (T358394).

Change 1006168 merged by jenkins-bot:

[mediawiki/core@master] phpunit: Do not crash when paratest options are passed in

https://gerrit.wikimedia.org/r/1006168

I've been investigating this and I have to say I gave up on paratest pretty fast because I wasn't able to convince it to run the right set of tests. I've implemented more or less the approach from Sebastian Bergmann and proposed a patch to quibble to try this out: https://gerrit.wikimedia.org/r/c/integration/quibble/+/1031903 . So far I'm pretty happy with the results, but I would very much welcome input and feedback from other people who are actively working on this.

There are a few tests that fail in the parallel runs, but I've been slowly working through those in T361190. Besides that, it would be good at some point in the future to have a nightly job that runs the linear suite and generates the .phpunit.result.cachefile so that Quibble can use that to move evenly distribute the test classes over the testing buckets. Of course that has the disadvantage that the ordering of the tests might change every night - with round-robin allocation there is at least a fair amount of stability about which tests end up running with which others.

I've been investigating this and I have to say I gave up on paratest pretty fast because I wasn't able to convince it to run the right set of tests.

I still use the ancient paratest (5.0.4) version that lets you specify a phpunit executable path. I run the dev_paratest wrapper script from https://github.com/AaronSchulz/mediawiki-developer-tools very often. It doesn't support the speed trap thing though. I sometimes get failures due to bad assumptions of test order and side-effects (basically bugs in our own tests).

Modern paratest versions must be installed together with phpunit via composer and share code AFAIK (thus tighter version requirements are needed and met by composer). Using modern paratest/PHPUnit runs into issues with our sloppy use of @dataProvider and static state all over the place and I get lots of failure spam (I changed a few providers and then gave up since it would take forever to fix).

It doesn't support the speed trap thing though.

This is still a thing in latest paratest AFAIK. There's this issue from a while ago: https://github.com/paratestphp/paratest/issues/771. OTOH, speedtrap seems unmaintained and the issue about PHPUnit 10 support has been open for almost 1.5 years now. We'll have to make some decision on it as part of T328919: Upgrade to PHPUnit 10 anyway.

Using modern paratest/PHPUnit runs into issues with our sloppy use of @dataProvider and static state all over the place and I get lots of failure spam (I changed a few providers and then gave up since it would take forever to fix).

That's odd, I don't remember seeing very many failures when experimenting with paratest. r1005792 is a POC for paratest in CI; while there are some failures, things do not seem to be hopeless. Unfortunately though, it's quite hard to experiment with this more thoroughly: firstly because of T345481; and secondly because of SQLite going rogue when tests are run in parallel (T50217#9111027), for which I think we'll need the code you have in mediawiki-developer-tools to be brought into core.

I've implemented more or less the approach from Sebastian Bergmann and proposed a patch to quibble to try this out: https://gerrit.wikimedia.org/r/c/integration/quibble/+/1031903 .

This patch looks good to me and I would be happy to see it deployed so we start to see reduction in build times.

Paratest implementation sounds nice for a variety of reasons (PHP based, external library, works locally) but until an implementation exists, I'd propose that we move forward with the Quibble patch. If/when we get Paratest working, we can revert the Quibble-based approach.

Change #1043719 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[integration/quibble@master] release: Quibble 1.9.0

https://gerrit.wikimedia.org/r/1043719