Page MenuHomePhabricator

PerformanceBudgetTest::testTotalModulesSize reported as risky test under PHPUnit 10
Open, Needs TriagePublic

Description

PHPUnit 10 reports a risky test with the reason "Test code or tested code printed unexpected output".

The config in phpunit.xml.template has failOnRisky="true", so this risky test, makes PHPUnit fail.

There was 1 risky test:

1) PerformanceBudgetTest::testTotalModulesSize
Test code or tested code printed unexpected output: Some tests raised warnings:
Total size of styles modules is 19.2kB ( 160.7 kB uncompressed).
If you are adding code on page load, please reduce styles that you are loading on page load.
Read https://www.mediawiki.org/wiki/Performance_budgeting for more context on this number.

Total size of scripts modules is 110.6kB ( 412.6 kB uncompressed).
If you are adding code on page load, please reduce scripts that you are loading on page load.
Read https://www.mediawiki.org/wiki/Performance_budgeting for more context on this number.

Related: 2f6871306cccb7101c0f35071b00c3cb24ed2ef6 / T391835: PerformanceBudgetTest should not print text in the middle of all MediaWiki PHPUnit runs

Event Timeline

Should we fail on risky? On PHPUnit 9 we warn but don't fail, right?

On both versions phpunit exit with code 1 in case of a risky test (due to failOnRisky="true") and that makes the CI job to fail.
There is also a setting beStrictAboutOutputDuringTests that is already true, it seems to cover now also the teardown function => that is a result of https://github.com/sebastianbergmann/phpunit/issues/5456

https://docs.phpunit.de/en/9.6/risky-tests.html
https://docs.phpunit.de/en/10.5/risky-tests.html