We use the @group Broken annotation to disable tests that shouldn't run by default, due to being known to fail, be flaky or otherwise undesirable to run in CI and locally by default for one reason or another.
In addition to this, we have @group Standalone as CI-specific optimization to run certain tests on direct patches but not when loaded from an upstream repository in a test for a downstream project.
That seems enough, but we have others that we exclude in various places. I'd like to question whether we actually still need to exclude those, and try to simplify this.
integration/config:/dockerfiles/quibble-coverage/mediawiki-coverage.sh
vendor/bin/phpunit \
--exclude-group Dump,Broken,ParserFuzz,Stubintegration/config:/dockerfiles/quibble-coverage/mwext-phpunit-coverage.sh
vendor/bin/phpunit \ --testsuite extensions:unit \ --exclude-group Dump,Broken,ParserFuzz,Stub
mediawiki/core:/includes/Composer/ComposerLaunchParallel.php
ALWAYS_EXCLUDE = [ 'Broken', 'ParserFuzz', 'Stub' ];
integration/quibble:/quibble/commands.py
always_excluded = ['Broken', 'ParserFuzz', 'Stub']
Evaluation
- Dump. This group is exclusive to MediaWiki core and is tagged on 6 test files (Codesearch). At glance, it sounds like a normal source code group (akin to @group ResourceLoader and @group BagOStuff) for Snapshot-related code, that we shouldn't be disabling by default. For example, ExportDemoTest validates the export-demo.xml file against the current schema. It seems to be what we would, today, call a "structure" test. It is marked "large", although it doesn't look large or slow. This should just be enabled? The good news is that is already removed from most of above files quoted above, but for some reason the coverage scripts are the only ones that exclude it. Is this a legacy left-over that we can sync away, or is there an undocumented reason to excluce this?
- Action: Sync and remove from CI config, or document and keep.
- ParserFuzz: This isn't used anymore by any test files. I'm guessing the (slow) fuzzTest.php script used to be implemented as a PHPUnit test. This isn't the case anymore.
- Action:: Remove from CI config?
- Stub: This group is exclusive to the Math extension today, and used by two test files (Codesearch): WikiTexVC/EnWikiFormulaeTest.php and WikiTexVC/ChemRegressionTest.php.
- Action: Replace with Broken or convert to a maintenance script.
- Action: Remove from CI config?