Change Details

# Background Test platform main goal this quarter is to reduce the developer feedback time for the core CI below 10 minutes [[ https://phabricator.wikimedia.org/T420590 | T420590 ]]. We are almost there, We now have all the jobs running below <10 minutes except `quibble-with-gated-extensions-vendor-mysql-php83`. # Problem In `quibble-with-gated-extensions-vendor-mysql-php83` job, we have noticed that the ResourcesTest class takes the most time to run by far, compared to other classes. Checking the [[ https://phpunit-results-cache.toolforge.org/results/quibble-with-gated-extensions-vendor-mysql-php83/37301 | cache results ]] of a sample run of the gated extensions job, the top 3 slowest classes are: ResourcesTest ~ 420 sec ParserTest_Cite_1_citeParserTests_Test ~98 sec ParserIntegrationTest ~50 sec When the PHPUnit test splitter distributes the classes to the available split groups, ResourcesTest gets assigned to one split group. In this [[ https://integration.wikimedia.org/ci/job/quibble-with-gated-extensions-vendor-mysql-php83/37301/ | sample case ]] case, the database section timings are like this: split_group_5 505.297s split_group_1 215.208s split_group_0 211.558s split_group_2 200.622s split_group_3 193.049s split_group_4 141.554s split_group_6 17.441s split_group_7 1.395s The ResourcesTest class is assigned to split group 5, together with 6 other classes. The below timings indicate how long the classes took: ResourcesTest 470.568s ApiStructureTest 20.383s ContentHandlerFunctionalTest 4.149s PerformanceBudgetTest 2.336s ApiPrefixUniquenessTest 0.231s DatabaseIntegrationTest 0.046s DumpableObjectsTest 0.032s Because the split groups run in parallel, the time taken by the database PHPUnit stage is decided by the slowest split group. The job has to wait for all split groups to finish before it can move on. This means the group that gets ResourcesTest is disproportionally slower than the other groups. Even though the other split groups are done, the job still has to wait for the ResourcesTest group to finish. # Exploration We have tried a few ideas to make this class run faster, but with our limited knowledge of the PHPunit tests and ResourceLoader, we have not been able to land a solution that does not introduce a maintenance burden to allow continuity. Examples of what we have tried: ([[ https://gerrit.wikimedia.org/r/c/mediawiki/core/+/1279380 | 1279380 ]], [[ https://gerrit.wikimedia.org/r/c/mediawiki/core/+/1284007 | 1284007 ]], [[ https://gerrit.wikimedia.org/r/c/mediawiki/core/+/1296081 | 1296081 ]]) # Solution If `ResourcesTest` can run faster, it would help us meet the goal we need. We are requesting input from anyone familiar with `ResourcesTest` and the ResourceLoader module. We want to reduce ResourcesTest runtime while keeping its guarantees.

# Background Test platform main goal this quarter is to reduce the developer feedback time for the core CI below 10 minutes [[ https://phabricator.wikimedia.org/T420590 | T420590 ]]. We are almost there, We now have all the jobs running below <10 minutes except `quibble-with-gated-extensions-vendor-mysql-php83`. # Problem In `quibble-with-gated-extensions-vendor-mysql-php83` job, we have noticed that the ResourcesTest class takes the most time to run by far, compared to other classes. Checking the [[ https://phpunit-results-cache.toolforge.org/results/quibble-with-gated-extensions-vendor-mysql-php83/37301 | cache results ]] of a sample run of the gated extensions job, the top 3 slowest test classes are: | Class | Duration |--|-- | ResourcesTest | ~ 420 sec | ParserTest_Cite_1_citeParserTests_Test | ~98 sec | ParserIntegrationTest | ~50 sec When the PHPUnit test splitter distributes the classes to the available split groups, `ResourcesTest` gets assigned to one split group. In this [[ https://integration.wikimedia.org/ci/job/quibble-with-gated-extensions-vendor-mysql-php83/37301/ | sample case ]] case, the database section timings are like this: | Group | Duration |--|-- | split_group_5 | **505.297s** | split_group_1 | 215.208s | split_group_0 | 211.558s | split_group_2 | 200.622s | split_group_3 | 193.049s | split_group_4 | 141.554s | split_group_6 | 17.441s | split_group_7 | 1.395s The `ResourcesTest` class is assigned to split group 5, together with 6 other classes. The below timings indicate how long the classes took inside that split group #5: | Test class | Duration |--|-- | ResourcesTest | 470.568s | ApiStructureTest | 20.383s | ContentHandlerFunctionalTest| 4.149s | PerformanceBudgetTest | 2.336s | ApiPrefixUniquenessTest | 0.231s | DatabaseIntegrationTest | 0.046s | DumpableObjectsTest | 0.032s Because the split groups run in parallel, the time taken by the database PHPUnit stage is decided by the slowest split group. The job has to wait for all split groups to finish before it can move on. This means **the group that gets `ResourcesTest` is disproportionally slower than the other groups**. Even though the other split groups are done, the job still has to wait for the ResourcesTest group to finish. # Exploration We have tried a few ideas to make this class run faster, but with our limited knowledge of the PHPunit tests, MediaWikiIntegrationTestCase and ResourceLoader, we have not been able to land a solution that does not introduce a maintenance burden to allow continuity. Examples of what we have tried: [[ https://gerrit.wikimedia.org/r/c/mediawiki/core/+/1279380 | 1279380 ]], //tests: reuse cache in ResourcesTest::testRespond// * The class crawls through each of the resource loader modules fed as a data provider to a test case. Profiling show a lot of time is spent in `Wikimedia\Minify\JavaScriptMinifier::minifyInternal`. That is for JavaScript minification which is a CPU intensive, the result is cached in `LocalServerObjectCache`. The setup/teardown wipes the minimification cache. * The change proposes to keep a copy of the Cache and restore it via setService(). Peter pointed that also reset all services which might be a problem? See https://gerrit.wikimedia.org/r/c/mediawiki/core/+/1279380/comments/3624ca64_af2e7a22 [[ https://gerrit.wikimedia.org/r/c/mediawiki/core/+/1284007 | 1284007 ]] //tests: reuse integration fixture in ResourcesTest::testRespond// * This is an attempt to flag a service to prevent it from being reset by the MediaWikiIntegrationTestCase setup/teardown. Antoine feels it is a lot of added complexity. [[ https://gerrit.wikimedia.org/r/c/mediawiki/core/+/1296081 | 1296081 ]]) //WIP: Split ResourceLoader response structure tests by module// * That split the large test to smaller batches. By balancing the tests to different split groups, the total feedback time is lowered. It is done by generating tests class rather than splitting them. * The cache is still wiped, thus the aggregated runtime is more or the same. # Goal If `ResourcesTest` can run faster, it would help us make the jobs run faster. We are requesting input from anyone familiar with `ResourcesTest`, MediaWikiIntegrationTestcase setup/teardown of services and the ResourceLoader module. We want to reduce ResourcesTest runtime while keeping its guarantees.

# Background Test platform main goal this quarter is to reduce the developer feedback time for the core CI below 10 minutes [[ https://phabricator.wikimedia.org/T420590 | T420590 ]]. We are almost there, We now have all the jobs running below <10 minutes except `quibble-with-gated-extensions-vendor-mysql-php83`. # Problem In `quibble-with-gated-extensions-vendor-mysql-php83` job, we have noticed that the ResourcesTest class takes the most time to run by far, compared to other classes. Checking the [[ https://phpunit-results-cache.toolforge.org/results/quibble-with-gated-extensions-vendor-mysql-php83/37301 | cache results ]] of a sample run of the gated extensions job, the top 3 slowest test classes are: ResourcesTest ~ 420 sec| Class | Duration ParserTest_Cite_1_citeParserTests_Test ~98 sec|--|-- | ResourcesTest | ~ 420 sec | ParserTest_Cite_1_citeParserTests_Test | ~98 sec | ParserIntegrationTest | ~50 sec When the PHPUnit test splitter distributes the classes to the available split groups, `ResourcesTest` gets assigned to one split group. In this [[ https://integration.wikimedia.org/ci/job/quibble-with-gated-extensions-vendor-mysql-php83/37301/ | sample case ]] case, the database section timings are like this: | Group | Duration |--|-- | split_group_5 | **505.297s** | split_group_1 | 215.208s | split_group_0 | 211.558s | split_group_2 | 200.622s | split_group_3 | 193.049s | split_group_4 | 141.554s | split_group_6 | 17.441s | split_group_7 | 1.395s The `ResourcesTest` class is assigned to split group 5, together with 6 other classes. The below timings indicate how long the classes took inside that split group #5: Resources| Test 470.568sclass | Duration ApiStructureTest 20.383s|--|-- ContentHandlerFunctional| ResourcesTest 4.149s| 470.568s PerformanceBudget| ApiStructureTest 2.336s| 20.383s ApiPrefixUniquenessTest 0.231s| ContentHandlerFunctionalTest| 4.149s DatabaseIntegration| PerformanceBudgetTest 0.046s| 2.336s DumpableObjects| ApiPrefixUniquenessTest | 0.231s | DatabaseIntegrationTest 0| 0.046s | DumpableObjectsTest | 0.032s Because the split groups run in parallel, the time taken by the database PHPUnit stage is decided by the slowest split group. The job has to wait for all split groups to finish before it can move on. This means **the group that gets `ResourcesTest is` is disproportionally slower than the other groups**. Even though the other split groups are done, the job still has to wait for the ResourcesTest group to finish. # Exploration We have tried a few ideas to make this class run faster, but with our limited knowledge of the PHPunit tests, MediaWikiIntegrationTestCase and ResourceLoader, we have not been able to land a solution that does not introduce a maintenance burden to allow continuity. Examples of what we have tried: ([[ https://gerrit.wikimedia.org/r/c/mediawiki/core/+/1279380 | 1279380 ]], [[ https://gerrit.wikimedia.org/r/c/mediawiki/core/+/1284007 | 1284007 ]], [[ https://gerrit.wikimedia.org/r/c/mediawiki/core/+/1296081 | 1296081 ]]) # Solution[[ https://gerrit.wikimedia.org/r/c/mediawiki/core/+/1279380 | 1279380 ]], //tests: reuse cache in ResourcesTest::testRespond// * The class crawls through each of the resource loader modules fed as a data provider to a test case. Profiling show a lot of time is spent in `Wikimedia\Minify\JavaScriptMinifier::minifyInternal`. That is for JavaScript minification which is a CPU intensive, the result is cached in `LocalServerObjectCache`. The setup/teardown wipes the minimification cache. * The change proposes to keep a copy of the Cache and restore it via setService(). Peter pointed that also reset all services which might be a problem? See https://gerrit.wikimedia.org/r/c/mediawiki/core/+/1279380/comments/3624ca64_af2e7a22 [[ https://gerrit.wikimedia.org/r/c/mediawiki/core/+/1284007 | 1284007 ]] //tests: reuse integration fixture in ResourcesTest::testRespond// * This is an attempt to flag a service to prevent it from being reset by the MediaWikiIntegrationTestCase setup/teardown. Antoine feels it is a lot of added complexity. [[ https://gerrit.wikimedia.org/r/c/mediawiki/core/+/1296081 | 1296081 ]]) //WIP: Split ResourceLoader response structure tests by module// * That split the large test to smaller batches. By balancing the tests to different split groups, the total feedback time is lowered. It is done by generating tests class rather than splitting them. * The cache is still wiped, thus the aggregated runtime is more or the same. # Goal If `ResourcesTest` can run faster, it would help us meetmake the goal we needjobs run faster. We are requesting input from anyone familiar with `ResourcesTest`, MediaWikiIntegrationTestcase setup/teardown of services and the ResourceLoader module. We want to reduce ResourcesTest runtime while keeping its guarantees.