Page MenuHomePhabricator

CI tests are failing because of PerformanceBudgetTest::testTotalModulesSize
Closed, ResolvedPublic1 Estimated Story Points

Description

The CI tests for Language Team's patches (probably for other teams too) constantly fail, because of the following error:

MediaWiki\Skins\Vector\Tests\Structure\PerformanceBudgetTest::testTotalModulesSize with data set "vector-2022" ('vector-2022', array('80KB', '17KB'))
18:37:02 T346813: Performance budget for styles in skin vector-2022 on main article namespace has been exceeded. Total size of styles modules is 17.2 Kbs is greater than the current budget size of 17 Kbs (see Vector/bundlesize.config.json).

The issue exists since at least 18 hours ago.

Example of a failing CI test: https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-php74-noselenium-docker/161818/console (and the related patch: https://gerrit.wikimedia.org/r/c/mediawiki/extensions/ContentTranslation/+/1010862/5)

02:37:05 1) MediaWiki\Skins\Vector\Tests\Structure\PerformanceBudgetTest::testTotalModulesSize with data set "vector-2022" ('vector-2022', array('80KB', '17KB'))
02:37:05 T346813: Performance budget for styles in skin vector-2022 on main article namespace has been exceeded. Total size of styles modules is 17.2 Kbs is greater than the current budget size of 17 Kbs (see Vector/bundlesize.config.json).
02:37:05 Please reduce styles that you are loading on page load or talk to the skin maintainer about modifying the budget.
02:37:05 The following modules are enabled on page load:
02:37:05 skins.vector.search.codex.styles
02:37:05 skins.vector.styles
02:37:05 skins.vector.user.styles
02:37:05 skins.vector.icons
02:37:05 ext.flaggedRevs.basic
02:37:05 ext.uls.pt
02:37:05 ext.visualEditor.desktopArticleTarget.noscript
02:37:05 site.styles
02:37:05 noscript
02:37:05 user.styles

TODO

  • Restore test
  • Investigate the source of the increase and how it bypassed CI.
  • Create tasks related to any potential regressions or follow up work if found.

Event Timeline

ngkountas triaged this task as Unbreak Now! priority.Mar 14 2024, 10:48 AM

Change 1011143 had a related patch set uploaded (by Jdlrobson; author: Jdlrobson):

[mediawiki/skins/Vector@master] Restore budget test with better guidance

https://gerrit.wikimedia.org/r/1011143

This test is not supposed to be skipped - it is supposed to act as a safe guard for performance so any increase should be investigated. I'll resolve once this has been restored.

I will investigate what and how this increased.

If it's not supposed to be skipped, what should we have done instead? The test failed on patches that didn't touch any RL modules and blocked our CI.

I've updated the instructions of the test on expected process. TDLR: You can increase the budget but it must always be investigated.

Jdlrobson lowered the priority of this task from Unbreak Now! to Medium.Mar 14 2024, 4:33 PM
Jdlrobson updated the task description. (Show Details)
KSarabia-WMF set the point value for this task to 1.Mar 14 2024, 4:54 PM

I think this is related to this patch: https://gerrit.wikimedia.org/r/c/mediawiki/skins/Vector/+/1010258 (T356677) which increased the amount of styles shipped by Vector 2022.

The issue here is FlaggedRevisions does not always run in CI and our developers don't have it installed. If installed the test fails.

My patch allocates extra space for FlaggedRevisions when detected since it is a special case.
I've also updated the test message to give better guidance on how to handle this in future.

Change 1011143 merged by jenkins-bot:

[mediawiki/skins/Vector@master] Restore budget test with better guidance and special case FlaggedRevs

https://gerrit.wikimedia.org/r/1011143

I chatted to @Jdforrester-WMF about this today and I think we've got the rough outline of a plan for moving this out of Vector. I'll create a ticket before week's end to document next steps.