Page MenuHomePhabricator

Run PHPUnit tests for mediawiki/core in parallel
Closed, ResolvedPublic

Description

Thanks to work done in T50217: Speed up MediaWiki PHPUnit build by running integration tests in parallel, T361118: [EPIC][Infra] Reduce CI test runtime for Wikibase and related extensions and other tasks, we now are able to run PHPUnit tests in parallel via Quibble in CI.

This has led to a significant improvement in build times, going from ~30 to ~10 minutes for the wmf-quibble-vendor-mysql-php74 job.

We should now look at enabling PHPUnit parallel execution for the mediawiki/core repo. Those still clock in regularly around the ~30 minute mark.

Enabling is straightforward:

if (
    # ... temporarily exclude MediaWiki core and extensions
    # that have issues with parallel tests
    params["ZUUL_PROJECT"] not in [
        "mediawiki/core",
        "mediawiki/extensions/WikiLambda",
        # DonationInterface uses a different branching model. Its master
        # branch is tested with mediawiki/core fundraising/REL1_39 branch
        # which does not have the parallel work.
        "mediawiki/extensions/DonationInterface",
    ]
    # ... exclude on REL_ branches (not yet tested/patched),
    and "ZUUL_BRANCH" in params
    and not params["ZUUL_BRANCH"].startswith("REL1")
    # Exclude fundraising branches and specific jobs
    and not params["ZUUL_BRANCH"].startswith("fundraising")
    and not job.name.startswith("quibble-fundraising")
):
    params['QUIBBLE_PHPUNIT_PARALLEL'] = '1'

and involves removing mediawiki/core from the list above in the integration/config repo. There will likely be some breakages when this is done, though, so we'd need to work out how to do this in a way that balances disruption and level of effort to avoid disruption. e.g. one approach might be to pick a Friday morning to remove mediawiki/core from the list, run some change through CI, gather the list of tests that break, then add back mediawiki/core to the config until we have time to update the tests.

Event Timeline

Enabling the environment variable is straightforward. But just enabling QUIBBLE_PHPUNIT_PARALLEL will still result in the extensions phpunit suite being run. I guess for core we want the default suite to run.

Enabling the environment variable is straightforward. But just enabling QUIBBLE_PHPUNIT_PARALLEL will still result in the extensions phpunit suite being run. I guess for core we want the default suite to run.

Should we have the default suite as a separate task, then? I think these are probably two distinct efforts.

I think I would keep them in the same task for now - it's going to be hard to test which suite runs without having the environment variable set to enable the parallel runs in the first place. If getting default running turns out to be a rabbit hole, we can then file a ticket for that.

I think I would keep them in the same task for now - it's going to be hard to test which suite runs without having the environment variable set to enable the parallel runs in the first place. If getting default running turns out to be a rabbit hole, we can then file a ticket for that.

Sounds good

Change #1093909 had a related patch set uploaded (by Arthur taylor; author: Arthur taylor):

[integration/config@master] Enable PHPUnit parallel testing for mediawiki/core

https://gerrit.wikimedia.org/r/1093909

I investigated this further. The wmf-quibble-core-vendor-mysql-php74 job (slow - 30 mins) actually runs the extensions suite, because --phpunit-testsuite=extensions is passed to Quibble for that job. Enabling parallel runs here should work without issue. For other jobs, for example mediawiki-quibble-vendor-mysql-php74 where the default suite is run, Quibble detects that the default suite is requested and falls back to a linear run - this is acceptable for now because the job only takes 10 mins as it is. We could look at a separate ticket to speed up mediawiki-quibble-vendor-mysql-php74 later.

In local testing of wmf-quibble-core-vendor-mysql-php74 and mediawiki-quibble-vendor-mysql-php74 with QUIBBLE_PHPUNIT_PARALLEL=1 set, everything seems to work without issue.

Change #1093909 merged by jenkins-bot:

[integration/config@master] Enable PHPUnit parallel testing for mediawiki/core

https://gerrit.wikimedia.org/r/1093909

Mentioned in SAL (#wikimedia-releng) [2024-11-25T10:02:22Z] <hashar> Reloaded Zuul to enable PHPUnit parallel testing for mediawiki/core | https://gerrit.wikimedia.org/r/1093909 | T379712

This work sped up the build process from T369115: [FY24-25 WE6.2.1] Publish pre-train single version containers significantly. Thank you!

[23:56]  <    bd808> kostajh: I just checked on the wmf/next build from last night. The parallel work on wmf-quibble-core-vendor-mysql-php74 helped a bunch. It ran 12 minutes faster than the historical average. We are now seeing wmf-quibble-selenium-php74 as the slowest suite (18m) with mediawiki-quibble-vendor-postgres-php74 close behind at 17m.
[23:56]  <    bd808> More at https://gerrit.wikimedia.org/r/c/mediawiki/core/+/1097575?tab=checks is you're interested.
[00:01]  <    bd808> *if
kostajh assigned this task to ArthurTaylor.

I think we can resolve this. Nice work @ArthurTaylor!