Vector PerformanceBudgetTest::testTotalModulesSize CI break
Closed, DuplicatePublic
Actions

Description

e.g. https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-php74-noselenium-docker/135427/console

21:53:45 There were 2 failures:
21:53:45 
21:53:45 1) MediaWiki\Skins\Vector\Tests\Structure\PerformanceBudgetTest::testTotalModulesSize with data set "vector-2022" ('vector-2022', array('68.1KB', '15KB'))
21:53:45 Performance budget for style in skin vector-2022 on main article namespace has been exceeded. Total size of style modules is 24718 bytes is greater than budget size 15360 bytes Reduce styles loaded on page load or talk to skin maintainer before modifying the budget.
21:53:45 Failed asserting that 24718 is equal to 15360.0 or is less than 15360.0.
21:53:45 
21:53:45 2) MediaWiki\Skins\Vector\Tests\Structure\PerformanceBudgetTest::testTotalModulesSize with data set "vector" ('vector', array('57.3KB', '8.3KB'))
21:53:45 Performance budget for style in skin vector on main article namespace has been exceeded. Total size of style modules is 17824 bytes is greater than budget size 8499.2 bytes Reduce styles loaded on page load or talk to skin maintainer before modifying the budget.
21:53:45 Failed asserting that 17824 is equal to 8499.2 or is less than 8499.2.

The test was added in T346813: Vector performance budget's should consider modules added by hooks.

Details

Subject	Repo	Branch	Lines +/-
Skip PerformanceBudgetTest::testTotalModulesSize	mediawiki/skins/Vector	master	+2 -0
Skip PerformanceBudgetTest::testTotalModulesSize	mediawiki/skins/Vector	wmf/1.42.0-wmf.4	+2 -0
Vector PerformanceBudgetTest::testTotalModulesSize CI break	mediawiki/skins/Vector	master	+42 -22
Skip PerformanceBudgetTest::testTotalModulesSize	mediawiki/skins/Vector	master	+2 -0
Restore test and increase budget with debug information	mediawiki/skins/Vector	master	+14 -7
Skip PerformanceBudgetTest::testTotalModulesSize	mediawiki/skins/Vector	master	+2 -0

Customize query in gerrit

Related Objects

Mentioned In: T365480: Investigate: Vector-2022 PerformanceBudgetTest blocks GrowthExperiments l10n change
T359679: Verify 1.42.0-wmf.23 deployment
T360590: Create a set of tests for defining a performance budget
T360102: CI tests are failing because of PerformanceBudgetTest::testTotalModulesSize
T350777: 1.42.0-wmf.4: Structured Data on Wikimedia Commons not longer available
T346813: Vector performance budget's should consider modules added by hooks
Mentioned Here: T350777: 1.42.0-wmf.4: Structured Data on Wikimedia Commons not longer available
T350514: Projects with FlaggedRevisions installed load the entire Codex library and OOUI and 10kb more CSS, over 500kb of JavaScript and likely have higher first paint
T346813: Vector performance budget's should consider modules added by hooks

Event Timeline

Tgr created this task.Nov 2 2023, 5:00 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 2 2023, 5:00 AM

At a blind guess, the test doesn't take into account styles added by other extensions via skinStyles.

Change 970906 had a related patch set uploaded (by Gergő Tisza; author: Gergő Tisza):

[mediawiki/skins/Vector@master] Skip PerformanceBudgetTest::testTotalModulesSize

https://gerrit.wikimedia.org/r/970906

gerritbot added a project: Patch-For-Review.Nov 2 2023, 5:03 AM

Change 970906 merged by jenkins-bot:

[mediawiki/skins/Vector@master] Skip PerformanceBudgetTest::testTotalModulesSize

https://gerrit.wikimedia.org/r/970906

Tgr updated the task description. (Show Details)Nov 2 2023, 5:25 AM

In T350338#9300447, @Tgr wrote:

At a blind guess, the test doesn't take into account styles added by other extensions via skinStyles.

Although I saw the CI error on CentralAuth patches and that extension doesn't use skin styles. Something something CI extension dependency chains?

Maintenance_bot removed a project: Patch-For-Review.Nov 2 2023, 5:30 AM

Fixed by disabling the test, for now.

ReleaseTaggerBot added a project: MW-1.42-notes (1.42.0-wmf.4; 2023-11-07).Nov 2 2023, 6:00 AM

Change 971508 had a related patch set uploaded (by Jdlrobson; author: Jdlrobson):

[mediawiki/skins/Vector@master] Restore test and increase budget with debug information

https://gerrit.wikimedia.org/r/971508

gerritbot added a project: Patch-For-Review.Nov 3 2023, 6:49 PM

Jdlrobson mentioned this in T346813: Vector performance budget's should consider modules added by hooks.Nov 3 2023, 6:53 PM

FYI: This was being caused by an existing issue the tests have flagged: T350514.

Change 971508 merged by jenkins-bot:

[mediawiki/skins/Vector@master] Restore test and increase budget with debug information

https://gerrit.wikimedia.org/r/971508

Maintenance_bot removed a project: Patch-For-Review.Nov 6 2023, 2:10 PM

WMDE-leszek subscribed.Nov 7 2023, 9:02 AM

Reopening as I am afraid it might be still happening. See eg. gate-and-submit jobs failing for https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Wikibase/+/971918 -- we're quite convinced this change cannot affect Resource Loader's module size.

See eg. from https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-php74-noselenium-docker/136171/console

09:46:41 1) MediaWiki\Skins\Vector\Tests\Structure\PerformanceBudgetTest::testTotalModulesSize with data set "vector-2022" ('vector-2022', array('70.1KB', '17.0KB'))
09:46:41 T346813: Performance budget for scripts in skin vector-2022 on main article namespace has been exceeded. Total size of script modules is 75521 bytes is greater than the current budget size of 71782.4 bytes Please reduce scripts that you are loading on page load or talk to the skin maintainer about modifying the budget. The following modules are enabled on page load:
09:46:41 site
09:46:41 mediawiki.page.ready
09:46:41 skins.vector.user
09:46:41 skins.vector.js
09:46:41 wikibase.vector.searchClient
09:46:41 ext.checkUser.clientHints
09:46:41 ext.echo.centralauth
09:46:41 ext.eventLogging
09:46:41 ext.uls.compactlinks
09:46:41 ext.uls.interface
09:46:41 ext.visualEditor.desktopArticleTarget.init
09:46:41 ext.visualEditor.targetLoader
09:46:41 wikibase.client.data-bridge.init
09:46:41 ext.wikimediaEvents
09:46:41 ext.wikimediaEvents.wikibase
09:46:41 user
09:46:41 user.options
09:46:41 Failed asserting that 75521 is equal to 71782.4 or is less than 71782.4.
09:46:41 
09:46:41 /workspace/src/skins/Vector/tests/phpunit/structure/PerformanceBudgetTest.php:181

I haven't looked at the test in detail but it does seem like its setup is quite fragile, in the sense it does not run for a fixed set of extensions and whatnot, but would run with an arbitrary budget number with whatever the instance running the test enabled?

Change 972329 had a related patch set uploaded (by WMDE-leszek; author: WMDE-leszek):

[mediawiki/skins/Vector@master] Skip PerformanceBudgetTest::testTotalModulesSize

https://gerrit.wikimedia.org/r/972329

gerritbot added a project: Patch-For-Review.Nov 7 2023, 9:13 AM

Change 972329 merged by jenkins-bot:

[mediawiki/skins/Vector@master] Skip PerformanceBudgetTest::testTotalModulesSize

https://gerrit.wikimedia.org/r/972329

ReleaseTaggerBot edited projects, added MW-1.42-notes (1.42.0-wmf.5; 2023-11-14); removed MW-1.42-notes (1.42.0-wmf.4; 2023-11-07).Nov 7 2023, 10:00 AM

Maintenance_bot removed a project: Patch-For-Review.Nov 7 2023, 10:10 AM

We've unblocked ourselves by skipping the test again. Apologies for the inconvenience.

Silvan_WMDE subscribed.Nov 7 2023, 10:34 AM

Change 971439 had a related patch set uploaded (by Mabualruz; author: Mabualruz):

[mediawiki/skins/Vector@master] Vector PerformanceBudgetTest::testTotalModulesSize CI break

https://gerrit.wikimedia.org/r/971439

gerritbot added a project: Patch-For-Review.Nov 7 2023, 4:47 PM

Change 971439 merged by jenkins-bot:

[mediawiki/skins/Vector@master] Vector PerformanceBudgetTest::testTotalModulesSize CI break

https://gerrit.wikimedia.org/r/971439

@WMDE-leszek sorry for the disruption here!

No worries about the inconvenience - we're expecting that by design as we're essentially retroactively documenting the current ecosystem so in future we can detect any significant changes in JS/CSS.

We've bumped the budget number for JS to consider Wikidata might be installed as this doesn't seem like a problematic amount of code (in other cases this test flagged existing issues/bugs with code shipping more than it should but that doesn't seem to apply here).

The intention is to define an explicit number for any install. On the long run we would like to avoid the installation of a single extension tipping us over that number.

Hopefully it won't happen, but if you encounter any false positives again please disable the test and ping us again.

Maintenance_bot removed a project: Patch-For-Review.Nov 7 2023, 7:10 PM

I think this check might work better as an alert than as a voting unit test. There are many extensions where Vector tests are not run on the extension's commits but some third unrelated extension will run Vector tests with the first extension installed, so the CI errors will show up in places completely unrelated to what caused them.

Thanks for the update and adjustment @Jdlrobson
I am likely missing to complete context but I think I am with @Tgr on suggesting to possibly considering some slightly different approach.
If the test is meant to be detecting significant changes in JS/CSS for WMF wikis, wouldn't it be more accurate to have tests - maybe even voting/blocking ones - that would run with the exact set of extensions and all as is provided on Wikipedia and so on? Possibly might be several "flavours" of such a check, as, as you notice, Wikidata runs a bit more code incl. JS/CSS for it custom UI, possibly similar situation could happen for Wikimedia Commons.
I don't know where such checks would live ideally in WMF Jenkins CI but having them as per-commit tests run on essentially any MW extension does seem like you might be tracking something slightly different that you actually intend to observe?

Change 972721 had a related patch set uploaded (by Jforrester; author: WMDE-leszek):

[mediawiki/skins/Vector@wmf/1.42.0-wmf.4] Skip PerformanceBudgetTest::testTotalModulesSize

https://gerrit.wikimedia.org/r/972721

gerritbot added a project: Patch-For-Review.Nov 8 2023, 4:10 PM

Change 972721 merged by jenkins-bot:

[mediawiki/skins/Vector@wmf/1.42.0-wmf.4] Skip PerformanceBudgetTest::testTotalModulesSize

https://gerrit.wikimedia.org/r/972721

Mentioned in SAL (#wikimedia-operations) [2023-11-08T16:39:33Z] <jforrester@deploy2002> Started scap: Backport for [[gerrit:972721|Skip PerformanceBudgetTest::testTotalModulesSize (T350338)]], [[gerrit:972720|Modify regex to reflect updated DOM (T350777)]]

Mentioned in SAL (#wikimedia-operations) [2023-11-08T16:40:56Z] <jforrester@deploy2002> jforrester: Backport for [[gerrit:972721|Skip PerformanceBudgetTest::testTotalModulesSize (T350338)]], [[gerrit:972720|Modify regex to reflect updated DOM (T350777)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)

Mentioned in SAL (#wikimedia-operations) [2023-11-08T16:47:03Z] <jforrester@deploy2002> Finished scap: Backport for [[gerrit:972721|Skip PerformanceBudgetTest::testTotalModulesSize (T350338)]], [[gerrit:972720|Modify regex to reflect updated DOM (T350777)]] (duration: 07m 29s)

ReleaseTaggerBot edited projects, added MW-1.42-notes (1.42.0-wmf.4; 2023-11-07); removed MW-1.42-notes (1.42.0-wmf.5; 2023-11-14).Nov 8 2023, 5:00 PM

Maintenance_bot removed a project: Patch-For-Review.Nov 8 2023, 5:10 PM

Change 1011080 had a related patch set uploaded (by Muhammad Jaziraly; author: Muhammad Jaziraly):

[mediawiki/skins/Vector@master] Skip PerformanceBudgetTest::testTotalModulesSize

https://gerrit.wikimedia.org/r/1011080

gerritbot added a project: Patch-For-Review.Mar 14 2024, 9:02 AM

Hello, It happened again and blocked CI for the Wikibase extension, so I disabled the test for now.

https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Wikibase/+/1011039?tab=checks
https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-php74-noselenium-docker/161802/console

Jakob_WMDE subscribed.Mar 14 2024, 9:09 AM

Change 1011080 merged by jenkins-bot:

[mediawiki/skins/Vector@master] Skip PerformanceBudgetTest::testTotalModulesSize

https://gerrit.wikimedia.org/r/1011080

Maintenance_bot removed a project: Patch-For-Review.Mar 14 2024, 10:30 AM

ReleaseTaggerBot edited projects, added MW-1.42-notes (1.42.0-wmf.23; 2024-03-19); removed MW-1.42-notes (1.42.0-wmf.4; 2023-11-07).Mar 14 2024, 1:00 PM

This test is not supposed to be skipped - it is supposed to act as a safe guard for performance so any increase should be investigated. I'll resolve once this has been restored.

I will investigate what and how this increased.

Jdlrobson added a project: Web-Team-Backlog (FY2023-24 Q3 Sprint 5).Mar 14 2024, 4:19 PM

Jdlrobson closed this task as a duplicate of T360102: CI tests are failing because of PerformanceBudgetTest::testTotalModulesSize.

In T350338#9631643, @Jdlrobson wrote:

This test is not supposed to be skipped - it is supposed to act as a safe guard for performance so any increase should be investigated. I'll resolve once this has been restored.

I will investigate what and how this increased.

If it's not supposed to be skipped, what should we have done instead? The test failed on patches that didn't touch any RL modules and blocked our CI.

Jdlrobson mentioned this in T360102: CI tests are failing because of PerformanceBudgetTest::testTotalModulesSize.Mar 14 2024, 4:31 PM

Jdlrobson mentioned this in T360590: Create a set of tests for defining a performance budget.Mar 21 2024, 12:56 AM

Jdlrobson added a project: Verified.Apr 1 2024, 12:51 AM

Edtadros mentioned this in T359679: Verify 1.42.0-wmf.23 deployment.Apr 3 2024, 9:55 PM

Michael mentioned this in T365480: Investigate: Vector-2022 PerformanceBudgetTest blocks GrowthExperiments l10n change.May 21 2024, 2:20 PM