I dug further into ResourcesTest::provideRespond() from the above data. Right now testRespond runs every ResourceLoader module against every installed skin. In the above job, the breakdown of the total 445 seconds taken by the ResourcesTest::testRespond is as follows:

vector-2022             80.654s
vector                  79.389s
minerva                 77.315s
contenttranslation      46.133s
json                    42.407s
authentication-popup    40.885s
apioutput               39.521s
fallback                39.206s

I guess my question would be if test coverage for all of these is important, or could we get away with limiting coverage to [ 'minerva', 'vector', 'vector-2022' ], or maybe what we could exempt?

Another idea we had tried was making ResourcesTest run its own split group. I took a sample gated-extensions job run, and went digging into the saved artifacts. I do not think we have a lot to gain by putting the ResourcesTest class in its own split group. For this particular run in the database section, the split group timings are like this:

split_group_5  505.297s
split_group_1  215.208s
split_group_0  211.558s
split_group_2  200.622s
split_group_3  193.049s
split_group_4  141.554s
split_group_6   17.441s
split_group_7    1.395s

The ResourcesTest class is assigned to split group 5, together with 6 other classes. The below timings indicate how long the classes took:

ResourcesTest                 470.568s
ApiStructureTest               20.383s
ContentHandlerFunctionalTest    4.149s
PerformanceBudgetTest           2.336s
ApiPrefixUniquenessTest         0.231s
DatabaseIntegrationTest         0.046s
DumpableObjectsTest             0.032s

ResourcesTest share is:

ResourcesTest time:  470.568 / 497.745s summed timed
Other class time:     27.177s

And the time ratios are consistent with other runs. Putting the class in it's own split group will save us ~30 seconds at most. This would not be a big diff, but 30 seconds is not enough to pursue this idea IMO, considering the job would still remain >10min mark.

Another approach is reusing the integration fixture in ResourcesTest::testRespond. ResourcesTest generates thousands of data-provider rows covering various test cases, and each row currently performs an expensive rebuild and teardown of the full integration fixture. The solution is to reuse the MediaWikiServices/DB fixture as described in the patchset. This saves time on every test scenario and cumulatively trims roughly 3 minutes off total runtime, bringing the gated extensions job below the 10 minute threshold. The implementation remains valid after we migrate to PHPUnit 10's upstream paratest. The savings are visible in the split group timings.

When tests run in parallel, the ResourcesTest class is assigned to a single split group, creating a bottleneck. One working approach is to split its heavy testRespond cases evenly across the 7 available split groups. The parallelization algorithm then treats each split as an independent test class, which lets us distribute them across the 7 split groups weighted by execution time. This balances the load more effectively and produces an immediate performance gain from the initial >370 sec run time of the class:

This does not reduce time. The extra step adds more time before the job finishes by two minutes. The better solution is to split the extension into its own job entirely.

@ArthurTaylor I eventually landed on the PHPUnit Results Cache tool and factored in the cached results when running the tests locally using Quibble, I had just forgotten to update the paste. The tool has also been very helpful with some other tasks we are working on as a team. I had done the logging implementation in Quibble, but was recommended to implement this in Core, so I am going to try that instead. I was not aware of the jenkins-run-analysis tool, thanks for the link. I will have a look at it :)

In T423059#11829443, @Peter wrote:

@pwangai do I understand that the splitting do not work good at all since some groups are really fast? One thing I was thinking was that since Wikibase tests is so in dominance, maybe we could hack so we run the tests in two steps: first all PHP for all others and then after they are finished the same with Wikibase. That way it would be easy to show the cost of the Wikibase tests. And maybe the split groups will work better if its Wikibase that takes the time?

I increased the count to 10 and ran multiple tests on a VPS instance and locally. With the already implemented algorithm for how the tests are split, increasing the count only yields diminishing returns and does not improve runtime, nor does it affect how the tests are distributed. The split group with the heaviest load still retains it, while the two additional split groups are assigned fast-running tests, causing them to finish in one or two seconds. Increasing split count is not effective in making the job run faster.

Removing the SlowTestDetector extension from monitoring and report generation saves about two seconds from the overall job runtime. However, I do not think that is enough time to justify sacrificing the reporting of slow tests in CI. The change would limit reporting to local runs during development and avoid generating reports in CI.

I was looking for multiple npm installs, like the one we had for selenium jobs, but mediawiki-core-vendor-mysql-php83 & quibble-with-gated-extensions-vendor-mysql-php83 jobs do run npm install once, which is necessary when running QUnit tests.

Sample summary once the tests finish:

With a bit of tinkering locally, I am able to extract some information indicating the number of loaded test files per repo and their distribution across the various split groups. A sample output at the beginning of the tests:

MediaWiki core tests do not run in parallel and instead run in serial. The biggest challenge to running the tests in parallel is that SQLite does not work well when it comes to database concurrency, eg ( T407954 ) This issue might be unique to our unit tests setup, as we are using a custom setup to run the tests in parallel. MySQL handles concurrency better, and a solution to speeding up the unit test job would be to run unit tests in parallel by default and fall back to running in serial whenever the --sqlite db args are detected. This will give us the benefit of faster runs and give us security whenever we run SQLite tests.

Closing this since there is a similar task in the works T419839

After some digging, I found during the setup process Quickstart adds workspaces ("workspaces": [".", "extensions/*", "skins/*"]) to mw core's package.json file and saves the original file as a backup but never restores the original after installation finishes. When working on an extension and you run "npm install" in the extension's directory, npm walks up the directory tree, discovers the root workspace config, and uses the root package-lock.json instead of resolving the extensions's dependencies independently. The old locked version wins, and your update is ignored. I will create a patch to restore the original package.json from the original backup that npm_workspaces already creates since workspace config was only needed during quickstart's install phase to load dependencies efficiently.