Thanks both, that makes sense. My bad on the paratest version, I had assumed I would be using the latest release when testing locally.

@Lars We are not limited to one db so we can have as many databases as we like. Paratest normally asigns a TEST_TOKEN to each worker, for example (1,2,3), and we can use that to point worker 1 at db1, worker 2 at db2 and so on. The only caveat I can see is we may need to slightly increase the size of tmpfs to something bigger than the current 1GB to accommodate the 6 workers. 2GB would be enough to cover that and have enough headroom for future data growth.

@ArthurTaylor In the original splitter logic, the last split group was reserved for ExtensionsParserTestSuite, which could not be listed because it generated its tests dynamically. Earlier this year that suite was removed in https://gerrit.wikimedia.org/r/c/mediawiki/core/+/1287963 (T422866: Migrate parser tests to new phpunit:config mechanism), and the parser tests now run within the normal 7 groups together with the other tests. The reservation itself stayed, and that split is still set aside but is never used to run any tests as described in T439227: Consistent 'no tests executed' for quibble split_group_7 . This unutilized compute could easily help our tests run faster if we used it. Do you think a change to allow the tests to utilize all groups is acceptable at this point? If yes, I can upload a patch to fix this for you to review.

Job quibble-with-gated-extensions-vendor-mysql-php83 run as Jenkins runs it, on agent integration-agent-docker-9998, 5 runs per cell, medians (min-max) in seconds. All 20 runs passed with 45,229 tests each.

My initial read, jenkins itself is not slower, the path to it is. The slowdown lines up with the services + traffic datacenter switchover on 2026-09-22 14:00 UTC.

I tested this on a ci agent, same image and arguments as the job, against current Quibble master.

I benchmarked 1289330. I Measured on a dedicated CI agent (10 CPUs) running the real quibble-for-mediawiki-core-vendor-sqlite-php83 job the way Jenkins runs it,
same image (quibble-bookworm-php83:1.21.0), same arguments, same castor cache, with the patched Quibble package bind mounted over the one in the image.

Duplicate of T432685

@Dzahn This will not involve sudo, I will be running a few scripts under my username to summarize the log data we need.

I would say this may not be a temporary one-off. Right now as a team we are formulating a KR revolving around castor improvements, and based on the current work, a need to access generated XML log files is necessary. Moving forward, we will keep observing castor performance, so continued access will be needed. Last quarter we were working on CI improvements and had several patches that needed deployment. We had help from Antoine from release engineering to do deployments.

Noted with thanks :)

SonarCloud Admin tag is now archived in QTE board, and the new tag is now reflects in Test Platform https://phabricator.wikimedia.org/project/board/8912/

@Peter Let us give it a try and collect feedback for a few days. We can then use the metrics we collect to inform our direction forward.

No worries @brennen . Thanks for your help so far. Once the jjb part is deployed, I will be around to test if the pipeline behaves as it should

mwext-codehealth-master-non-voting has the biggest cache size. In May, the cache distribution in size was as follows:

We are migrating to upstream parallelization, so I do not think we need to put much effort into this.

I do not have access to https://gitlab.wikimedia.org/repos/maven/wmf-jvm-parent-pom, but I have sent an access request. Someone else might have to make the change.

Much appreciated, @Krinkle. The latest gated extensions run (F88553062) brings the class down to ~200 sec from the previous >450 sec, a big improvement! Marking this as resolved.

I dug further into ResourcesTest::provideRespond() from the above data. Right now testRespond runs every ResourceLoader module against every installed skin. In the above job, the breakdown of the total 445 seconds taken by the ResourcesTest::testRespond is as follows:

vector-2022             80.654s
vector                  79.389s
minerva                 77.315s
contenttranslation      46.133s
json                    42.407s
authentication-popup    40.885s
apioutput               39.521s
fallback                39.206s

I guess my question would be if test coverage for all of these is important, or could we get away with limiting coverage to [ 'minerva', 'vector', 'vector-2022' ], or maybe what we could exempt?

Another idea we had tried was making ResourcesTest run its own split group. I took a sample gated-extensions job run, and went digging into the saved artifacts. I do not think we have a lot to gain by putting the ResourcesTest class in its own split group. For this particular run in the database section, the split group timings are like this:

split_group_5  505.297s
split_group_1  215.208s
split_group_0  211.558s
split_group_2  200.622s
split_group_3  193.049s
split_group_4  141.554s
split_group_6   17.441s
split_group_7    1.395s

The ResourcesTest class is assigned to split group 5, together with 6 other classes. The below timings indicate how long the classes took:

ResourcesTest                 470.568s
ApiStructureTest               20.383s
ContentHandlerFunctionalTest    4.149s
PerformanceBudgetTest           2.336s
ApiPrefixUniquenessTest         0.231s
DatabaseIntegrationTest         0.046s
DumpableObjectsTest             0.032s

ResourcesTest share is:

ResourcesTest time:  470.568 / 497.745s summed timed
Other class time:     27.177s

And the time ratios are consistent with other runs. Putting the class in it's own split group will save us ~30 seconds at most. This would not be a big diff, but 30 seconds is not enough to pursue this idea IMO, considering the job would still remain >10min mark.