Page MenuHomePhabricator
Create Task
Maniphest T345481

Migrate Parser and extension tests away from deprecated PHPUnit TestSuite subclassing
Open, Needs TriagePublic
Actions

Description

We currently use "dynamic" test suites for parser tests, as well as extension/skin tests (including unit tests after r937559). On a theoretical level, this is not ideal because the list of tests to execute cannot be determined immediately, but is generated dynamically after PHPUnit has finished its setup. In practice, these dynamic test suites make it impossible to use paratest (T50217), which is unable to split the tests across multiple processes. On top of that extending TestSuite is no longer possible in PHPUnit 10 (T328919).

Therefore, we need to find another way to generate these test lists dynamically.

  • Extension tests suite
  • Parser tests suite

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
commands: Run composer phpunit:config for parallel testsintegration/quibblemaster+8 -0
Generate local PHPUnit config before preparing parallel runsmediawiki/coremaster+2 -0
jjb: Avoid `composer phpunit:config` invocation in non-MW reposintegration/configmaster+0 -5
jjb: Generate PHPUnit config before running testsintegration/configmaster+9 -4
tests: Fix QueueingRemediatorTest requestId mismatchmediawiki/extensions/CirrusSearchmaster+9 -1
Rename phpunit.xml.dist to phpunit.xml.templatemediawiki/coremaster+40 -27
phpunit: Add check for local config in bootstrap.phpmediawiki/coremaster+16 -0
dockerfiles: [quibble-coverage] Use local PHPUnit configintegration/configmaster+15 -4
phpunit: Merge `skins` suite into `extensions`mediawiki/coremaster+8 -27
Introduce a CLI script to generate PHPUnit configmediawiki/coremaster+133 -110
Customize query in gerrit

Related Objects
Search...

Event Timeline

Daimona created this task.Sep 1 2023, 10:38 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 1 2023, 10:38 PM
Daimona mentioned this in T328919: Upgrade PHPUnit in MediaWiki code to PHPUnit 10.Sep 1 2023, 10:39 PM
Daimona added parent tasks: T50217: Speed up MediaWiki PHPUnit build by running integration tests in parallel, T328919: Upgrade PHPUnit in MediaWiki code to PHPUnit 10.
Novem_Linguae subscribed.Sep 1 2023, 10:44 PM
Daimona mentioned this in T347407: SonarCloud not picking up unit test coverage on master and patch branches of extensions and core.Oct 23 2023, 8:15 PM
Krinkle renamed this task from Stop extending PHPUnit's TestSuite class to Migrate Parser tests away from deprecated PHPUnit TestSuite subclassing.Dec 14 2023, 10:59 PM
Krinkle moved this task from Inbox to PHPUnit on the MediaWiki-Core-Tests board.
Krinkle added projects: MediaWiki-Parser, Content-Transform-Team.
MSantos edited projects, added Content-Transform-Team-WIP, Essential-Work; removed Content-Transform-Team.Dec 21 2023, 3:09 PM
Daimona renamed this task from Migrate Parser tests away from deprecated PHPUnit TestSuite subclassing to Migrate Parser and extension tests away from deprecated PHPUnit TestSuite subclassing.Jan 28 2024, 6:25 PM

I've updated the task title to clarify that, unfortunately, it's not only parser tests that subclass TestSuite, but also extension/skin tests...

Daimona mentioned this in T358097: composer phpunit hangs forever on WSL when running inside docker.Feb 21 2024, 12:48 PM
gerritbot added a comment.Feb 22 2024, 12:26 AM

Change 1005636 had a related patch set uploaded (by Daimona Eaytoy; author: Daimona Eaytoy):

[mediawiki/core@master] [POC] Introduce a CLI script to generate PHPUnit config

https://gerrit.wikimedia.org/r/1005636

gerritbot added a project: Patch-For-Review.Feb 22 2024, 12:26 AM
Daimona added subscribers: pmiazga, daniel, Krinkle.Feb 22 2024, 12:32 AM

My current proposal is to introduce a new CLI script that loads LocalSettings, and then uses the list of loaded extensions to generate a local phpunit.xml file, which can then be used to run tests normally. I've implemented this in the above proof of concept. CC'ing @Krinkle, @pmiazga, @daniel since we discussed this earlier today, in case you'd like to take a look and provide feedback. If not, apologies and feel free to unsubscribe :)

Daimona added a comment.Feb 23 2024, 12:30 AM

I also realized that there are more dynamic suites in extensions. These also need to be migrated. If we choose the approach above (script to autogenerate PHPUnit config), we may allow extensions to hook into the script and add/change suites; however, this adds complexity. It would be better to use other solutions in extensions, if possible.

Daimona mentioned this in T358394: Stop using internal PHPUnit code in Scribunto.Feb 23 2024, 9:55 PM
kostajh subscribed.Feb 26 2024, 8:28 AM
Daimona mentioned this in T50217: Speed up MediaWiki PHPUnit build by running integration tests in parallel.Feb 26 2024, 12:53 PM
pwangai subscribed.Feb 27 2024, 7:49 PM
Daimona added a project: User-Daimona.Mar 3 2024, 5:18 PM
Daimona moved this task from Backlog to Doing on the User-Daimona board.
ArthurTaylor mentioned this in T379764: phpunit splitter fails with obscure error message when a data provider is invalid.Nov 15 2024, 1:52 PM
Pppery edited projects, added Patch-Needs-Improvement; removed Patch-For-Review.Dec 17 2024, 3:34 AM
ihurbain edited projects, added Content-Transform-Team; removed Content-Transform-Team-WIP.Apr 11 2025, 3:45 PM
cscott mentioned this in T315768: Can we make parser tests show as PHP test coverage?.Apr 11 2025, 6:14 PM
cscott added a parent task: T315768: Can we make parser tests show as PHP test coverage?.
cscott mentioned this in T358955: Interesting code coverage in composer-coverage-patch-docker.
Physikerwelt subscribed.Apr 15 2025, 10:36 PM
cscott moved this task from Backlog to Tracking on the Content-Transform-Team board.May 1 2025, 2:43 PM
Daimona moved this task from Doing to Backlog on the User-Daimona board.Jun 1 2025, 12:34 AM
Daimona mentioned this in T418217: wikibase-client-php83 CI fails: Class "LuaStandalone: Wikibase\Client\Tests\Integration\DataAccess\Scribunto\WikibaseEntityLibraryTest" does not exist.Feb 24 2026, 4:59 PM
Daimona edited projects, added Patch-For-Review; removed Patch-Needs-Improvement.Mar 3 2026, 3:39 PM
Daimona added a comment.Mar 3 2026, 3:52 PM

For extension/skin tests, I have updated my previous patch and marked it as ready for review. The approach is to add an extra "build" step to PHPUnit, so you would run composer phpunit:config before running the tests (and note that this is only necessary if the set of available extensions/test directories changes). The composer phpunit command also does that for you automatically. Then we have a new structure test to verify that the config is up-to-date.

That would leave parser tests as the only remaining usage of dynamic suites, now that Scribunto tests have been updated in T358394. However, I'm not familiar with parser tests and don't know what the best solution would be for them. I can provide help if needed, but I can't drive this change because I don't have the necessary context.

gerritbot added a comment.Mar 3 2026, 8:19 PM

Change #1005636 merged by jenkins-bot:

[mediawiki/core@master] Introduce a CLI script to generate PHPUnit config

https://gerrit.wikimedia.org/r/1005636

ReleaseTaggerBot added a project: MW-1.46-notes (1.46.0-wmf.19; 2026-03-10).Mar 3 2026, 9:00 PM
gerritbot added a comment.Mar 3 2026, 9:03 PM

Change #1247681 had a related patch set uploaded (by Daimona Eaytoy; author: Daimona Eaytoy):

[mediawiki/core@master] phpunit: Merge `skins` suite into `extensions`

https://gerrit.wikimedia.org/r/1247681

Jdforrester-WMF updated the task description. (Show Details)Mar 3 2026, 9:21 PM
gerritbot added a comment.Mar 3 2026, 9:27 PM

Change #1247681 merged by jenkins-bot:

[mediawiki/core@master] phpunit: Merge `skins` suite into `extensions`

https://gerrit.wikimedia.org/r/1247681

Jdforrester-WMF added subscribers: cscott, Jdforrester-WMF.Mar 3 2026, 10:40 PM

CCing @cscott as the wizard for thorny Parser test runner issues.

gerritbot added a comment.Mar 4 2026, 12:55 AM

Change #1247719 had a related patch set uploaded (by Daimona Eaytoy; author: Daimona Eaytoy):

[mediawiki/core@master] phpunit: Add check for local config in bootstrap.php

https://gerrit.wikimedia.org/r/1247719

gerritbot added a comment.Mar 4 2026, 9:46 PM

Change #1248112 had a related patch set uploaded (by Daimona Eaytoy; author: Daimona Eaytoy):

[integration/config@master] dockerfiles: [quibble-coverage] Use local PHPUnit config

https://gerrit.wikimedia.org/r/1248112

gerritbot added a comment.Mar 4 2026, 9:54 PM

Change #1248113 had a related patch set uploaded (by Daimona Eaytoy; author: Daimona Eaytoy):

[integration/config@master] jjb: Generate PHPUnit config before running tests

https://gerrit.wikimedia.org/r/1248113

Dreamy_Jazz subscribed.Mar 4 2026, 10:08 PM

Not sure if this has caused T419073: mwext-phpunit-coverage-patch no longer runs any tests for modified files?

gerritbot added a comment.Mar 4 2026, 10:13 PM

Change #1248119 had a related patch set uploaded (by Daimona Eaytoy; author: Daimona Eaytoy):

[mediawiki/core@master] Rename phpunit.xml.dist to phpunit.xml.template

https://gerrit.wikimedia.org/r/1248119

Daimona mentioned this in T419073: mwext-phpunit-coverage-patch no longer runs any tests for modified files.Mar 4 2026, 10:34 PM
gerritbot added a comment.Mar 5 2026, 12:01 AM

Change #1248112 merged by jenkins-bot:

[integration/config@master] dockerfiles: [quibble-coverage] Use local PHPUnit config

https://gerrit.wikimedia.org/r/1248112

Stashbot added a comment.Mar 5 2026, 12:04 AM

Mentioned in SAL (#wikimedia-releng) [2026-03-05T00:04:36Z] <James_F> Docker: [quibble-coverage] Use local PHPUnit config, for T345481

gerritbot added a comment.Mar 5 2026, 12:50 AM

Change #1247719 merged by jenkins-bot:

[mediawiki/core@master] phpunit: Add check for local config in bootstrap.php

https://gerrit.wikimedia.org/r/1247719

gerritbot added a comment.Mar 5 2026, 1:09 AM

Change #1248113 merged by jenkins-bot:

[integration/config@master] jjb: Generate PHPUnit config before running tests

https://gerrit.wikimedia.org/r/1248113

gerritbot added a comment.Mar 5 2026, 1:26 AM

Change #1248119 merged by jenkins-bot:

[mediawiki/core@master] Rename phpunit.xml.dist to phpunit.xml.template

https://gerrit.wikimedia.org/r/1248119

gerritbot added a comment.Mar 5 2026, 8:27 AM

Change #1248394 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[mediawiki/extensions/CirrusSearch@master] tests: Fix QueueingRemediatorTest requestId mismatch

https://gerrit.wikimedia.org/r/1248394

gerritbot added a comment.Mar 5 2026, 8:52 AM

Change #1248400 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[integration/quibble@master] commands: Run composer phpunit:config for parallel tests

https://gerrit.wikimedia.org/r/1248400

kostajh mentioned this in T419107: The PHPUnit config override does not appear to be auto-generated.Mar 5 2026, 8:58 AM
gerritbot added a comment.Mar 5 2026, 12:18 PM

Change #1248452 had a related patch set uploaded (by Daimona Eaytoy; author: Daimona Eaytoy):

[mediawiki/core@master] Generate local PHPUnit config before preparing parallel runs

https://gerrit.wikimedia.org/r/1248452

Daimona mentioned this in T419113: QueueingRemediatorTest::testJobIsSent fails due to non-deterministic requestId.Mar 5 2026, 12:25 PM
gerritbot added a comment.Mar 5 2026, 1:46 PM

Change #1248394 merged by jenkins-bot:

[mediawiki/extensions/CirrusSearch@master] tests: Fix QueueingRemediatorTest requestId mismatch

https://gerrit.wikimedia.org/r/1248394

Daimona mentioned this in T419122: UpdateEmergencyDefaultsTest::testUpdateOK: Failed asserting that true is an instance of class "stdClass"..Mar 5 2026, 2:18 PM
gerritbot added a comment.Mar 5 2026, 2:38 PM

Change #1248452 merged by jenkins-bot:

[mediawiki/core@master] Generate local PHPUnit config before preparing parallel runs

https://gerrit.wikimedia.org/r/1248452

gerritbot added a comment.Mar 5 2026, 2:45 PM

Change #1248500 had a related patch set uploaded (by Daimona Eaytoy; author: Daimona Eaytoy):

[integration/config@master] jjb: Avoid `composer phpunit:config` invocation in non-MW repos

https://gerrit.wikimedia.org/r/1248500

gerritbot added a comment.Mar 5 2026, 2:54 PM

Change #1248500 merged by jenkins-bot:

[integration/config@master] jjb: Avoid `composer phpunit:config` invocation in non-MW repos

https://gerrit.wikimedia.org/r/1248500

gerritbot added a comment.Mar 6 2026, 12:48 PM

Change #1248400 abandoned by Hashar:

[integration/quibble@master] commands: Run composer phpunit:config for parallel tests

Reason:

Thank you!

https://gerrit.wikimedia.org/r/1248400

Daimona mentioned this in T407954: PHPUnit parallel database tests can fail to acquire locks on the same page.Mar 9 2026, 8:55 PM
cscott created subtask T422866: Migrate parser tests to new phpunit:config mechanism.Thu, Apr 9, 6:45 PM

I created a subtask, T422866: Migrate parser tests to new phpunit:config mechanism, for the parser test-related piece of this. CTT isn't likely to get to this before next quarter though.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits