Page MenuHomePhabricator

Flaky QUnit tests: Firefox 115 Script error
Closed, ResolvedPublic

Description

We (Growth-Team) are noticing that QUnit tests are failing in a flaky way more often recently.

The logs are not very clear as to what the cause is. The last test that is mentioned in the logs seems to be always from Visual Editor, but the specifics seem to change.

The relevant part from a recent (Feb 10) example:

00:06:37.473   ve.ui.WindowAction
00:06:37.473     โœ” Basic open/close
00:06:37.477   ve.ui.DataTransferHandlerFactory
00:06:37.477     โœ” getHandlerNameForItem
00:06:37.494   ve.ui.DSVFileTransferHandler
00:06:37.494     โœ” getInsertableData
00:06:37.517   ve.ui.UrlStringTransferHandler
00:06:37.517     โœ” paste
00:06:37.530 Firefox 115.0 (Linux x86_64) ERROR
00:06:37.530   Script error.
00:06:37.531   at http://127.0.0.1:9413/load.php?modules=%40wikimedia%2Fcodex%2Ccodex-search-styles%2Ccodex-styles%2Ccolor-picker%2CdataValues%2CdiffMatchPatch%2Cdompurify%2Cmapbox%7CdataValues.DataValue%2CTimeValue%2Cvalues%7Cext.articleplaceholder.createArticle%7Cext.centralauth.ForeignApi%7Cext.checkUser%2CeventLogging%2Cflow%2CglobalBlocking%2Cthanks%2CwikimediaEvents%7Cext.checkUser.clientHints%7Cext.cite.parsoid.styles%7Cext.cite.referencePreviews%2Cstyles%2Cux-enhancements%2CvisualEditor%7Cext.cite.visualEditor.core%7Cext.discussionTools.init%2Cminervaicons%7Cext.discussionTools.init.styles%7Cext.echo.api%2Cdm%2Cinit%2Cmobile%2Csecondaryicons%2Cui%7Cext.eventLogging.debug%7Cext.flow.components%2Ctemplating%2Cui%7Cext.flow.jquery.conditionalScroll%2CfindWithParent%7Cext.flow.visualEditor.icons%7Cext.graph.render%2CvisualEditor%7Cext.growthExperiments.DataStore%2CPostEdit%2CSuggestedEditSession%2Cicons%7Cext.growthExperiments.Homepage.SuggestedEdits%7Cext.guidedTour.lib%2Cstyles%7Cext.guidedTour.lib.internal%7Cext.kartographer.dialog%7Cext.popups.main%7Cext.securepoll.htmlform%7Cext.templateDataGenerator.data%7Cext.uls.common%2Ci18n%2Clanguagenames%2Cmediawiki%2Cmessages%2Cpreferences%2Cwebfonts%7Cext.visualEditor.data%2Cicons-licenses%2CmoduleIcons%2CmoduleIndicators%7Cjquery.animateWithEvent%2Cclient%2ChighlightText%2Ci18n%2Cinputautoexpand%2ClengthLimit%2CmakeCollapsible%2Cspinner%2Ctablesorter%2CtextSelection%2Cui%2Culs%2Cvalueview%7Cjquery.event.special.eachchange%7Cjquery.makeCollapsible.styles%7Cjquery.spinner.styles%7Cjquery.tablesorter.styles%7Cjquery.ui.commonssuggester%2Clanguagesuggester%2Csuggester%7Cjquery.uls.data%2Cgrid%7Cjquery.util.getDirectionality%7Cjquery.valueview.Expert%2CExpertExtender%2CExpertStore%7Cjquery.valueview.experts.CommonsMediaType%2CEmptyValue%2CGeoShape%2CGlobeCoordinateInput%2CMonolingualText%2CQuantityInput%2CStringValue%2CTabularData%2CTimeInput%2CUnDeserializableValue%7Cjquery.wikibase.entityselector%2Cwbtooltip%7Cjquery.wikibase.toolbar.styles&version=16w1x:0:0
00:06:37.537 10 02 2025 16:17:23.324:DEBUG [Firefox 115.0 (Linux x86_64)]: EXECUTING -> CONNECTED
00:06:37.545 10 02 2025 16:17:23.332:DEBUG [launcher]: CAPTURED -> BEING_KILLED
00:06:37.546 10 02 2025 16:17:23.332:DEBUG [launcher]: BEING_KILLED -> BEING_FORCE_KILLED
00:06:37.546 10 02 2025 16:17:23.334:DEBUG [Firefox 115.0 (Linux x86_64)]: CONNECTED -> DISCONNECTED
00:06:37.549 
00:06:37.549 Finished in 58.381 secs / 56.736 secs @ 16:17:23 GMT+0000 (Coordinated Universal Time)
00:06:37.549 
00:06:37.549 SUMMARY:
00:06:37.549 โœ” 2245 tests completed
00:06:37.549 โ„น 19 tests skipped
00:06:37.550 10 02 2025 16:17:23.337:DEBUG [karma-server]: Run complete, exiting.
00:06:37.551 10 02 2025 16:17:23.338:DEBUG [launcher]: Disconnecting all browsers
00:06:37.551 10 02 2025 16:17:23.339:DEBUG [launcher]: BEING_FORCE_KILLED -> BEING_FORCE_KILLED
00:06:37.551 10 02 2025 16:17:23.339:DEBUG [proxy]: Destroying proxy agents
00:06:37.584 10 02 2025 16:17:23.371:DEBUG [launcher]: Process FirefoxHeadless exited with code null and signal SIGTERM
00:06:37.585 10 02 2025 16:17:23.372:DEBUG [temp-dir]: Cleaning temp dir /tmp/karma-64727031
00:06:37.793 10 02 2025 16:17:23.580:DEBUG [launcher]: Finished all browsers
00:06:37.793 10 02 2025 16:17:23.580:DEBUG [launcher]: BEING_FORCE_KILLED -> FINISHED
00:06:37.793 10 02 2025 16:17:23.581:DEBUG [launcher]: FINISHED -> FINISHED
00:06:37.796 Warning: Task "karma:firefox" failed. Use --force to continue.
00:06:37.796 
00:06:37.797 Aborted due to warnings.

https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-php74-noselenium/72941/consoleFull

Event Timeline

Restricted Application added a subscriber: Aklapper. ยท View Herald TranscriptFeb 10 2025, 6:16 PM

Happening after a different VE test:

00:04:20.466   ve.ui.MWParameterCheckboxInputWidget
00:04:20.466     โœ” Constructor passes config to parent
00:04:20.468     โœ” setValue( "1" )
00:04:20.470     โœ” setValue( "0" )
00:04:20.472     โœ” setValue( empty string )
00:04:20.474     โœ” setValue( unexpected string )
00:04:20.476     โœ” setValue( unexpected type )
00:04:20.480     โœ” setSelected( true )
00:04:20.480     โœ” setSelected( false )
00:04:20.489   ve.ui.MWTemplateTitleInputWidget
00:04:20.490     โœ” default prefixsearch
00:04:20.499     โœ” CirrusSearch: all API parameters
00:04:20.504     โœ” CirrusSearch: showRedirectTargets disabled
00:04:20.510     โœ” CirrusSearch: prefixsearch behavior [0]
00:04:20.519     โœ” CirrusSearch: prefixsearch behavior [1]
00:04:20.523     โœ” CirrusSearch: prefixsearch behavior [2]
00:04:20.528     โœ” CirrusSearch: prefixsearch behavior [3]
00:04:20.545     โœ” CirrusSearch: prefixsearch behavior [4]
00:04:20.546     โœ” CirrusSearch: prefixsearch behavior [5]
00:04:20.546     โœ” CirrusSearch: prefixsearch behavior [6]
00:04:20.546     โœ” CirrusSearch: prefixsearch behavior [7]
00:04:20.552     โœ” CirrusSearch: prefixsearch behavior [8]
00:04:20.556     โœ” CirrusSearch: prefixsearch behavior [9]
00:04:20.561     โœ” CirrusSearch: prefixsearch behavior [10]
00:04:20.565     โœ” CirrusSearch: prefixsearch behavior [11]
00:04:20.570     โœ” CirrusSearch: prefixsearch behavior [12]
00:04:20.574     โœ” CirrusSearch: prefixsearch behavior [13]
00:04:20.578     โœ” CirrusSearch: prefixsearch behavior [14]
00:04:20.583     โœ” CirrusSearch: prefixsearch behavior [15]
00:04:20.588     โœ” CirrusSearch: prefixsearch behavior [16]
00:04:20.595     โœ” CirrusSearch: prefixsearch behavior [17]
00:04:20.595     โœ” CirrusSearch: prefixsearch behavior [18]
00:04:20.600     โœ” CirrusSearch: prefixsearch behavior [19]
00:04:20.630 Firefox 115.0 (Linux x86_64) ERROR
00:04:20.630   Script error.
00:04:20.631   at http://127.0.0.1:9413/load.php?modules=%40wikimedia%2Fcodex%2Ccodex-search-styles%2Ccodex-styles%2Ccolor-picker%2CdataValues%2CdiffMatchPatch%2Cdompurify%2Cmapbox%7CdataValues.DataValue%2CTimeValue%2Cvalues%7Cext.articleplaceholder.createArticle%7Cext.centralauth.ForeignApi%7Cext.checkUser%2CeventLogging%2Cflow%2CglobalBlocking%2Cthanks%2CwikimediaEvents%7Cext.checkUser.clientHints%7Cext.cite.parsoid.styles%7Cext.cite.referencePreviews%2Cstyles%2Cux-enhancements%2CvisualEditor%7Cext.cite.visualEditor.core%7Cext.discussionTools.init%2Cminervaicons%7Cext.discussionTools.init.styles%7Cext.echo.api%2Cdm%2Cinit%2Cmobile%2Csecondaryicons%2Cui%7Cext.eventLogging.debug%7Cext.flow.components%2Ctemplating%2Cui%7Cext.flow.jquery.conditionalScroll%2CfindWithParent%7Cext.flow.visualEditor.icons%7Cext.graph.render%2CvisualEditor%7Cext.growthExperiments.DataStore%2CPostEdit%2CSuggestedEditSession%2Cicons%7Cext.growthExperiments.Homepage.SuggestedEdits%7Cext.guidedTour.lib%2Cstyles%7Cext.guidedTour.lib.internal%7Cext.kartographer.dialog%7Cext.popups.main%7Cext.securepoll.htmlform%7Cext.templateDataGenerator.data%7Cext.uls.common%2Ci18n%2Clanguagenames%2Cmediawiki%2Cmessages%2Cpreferences%2Cwebfonts%7Cext.visualEditor.data%2Cicons-licenses%2CmoduleIcons%2CmoduleIndicators%7Cjquery.animateWithEvent%2Cclient%2ChighlightText%2Ci18n%2Cinputautoexpand%2ClengthLimit%2CmakeCollapsible%2Cspinner%2Ctablesorter%2CtextSelection%2Cui%2Culs%2Cvalueview%7Cjquery.event.special.eachchange%7Cjquery.makeCollapsible.styles%7Cjquery.spinner.styles%7Cjquery.tablesorter.styles%7Cjquery.ui.commonssuggester%2Clanguagesuggester%2Csuggester%7Cjquery.uls.data%2Cgrid%7Cjquery.util.getDirectionality%7Cjquery.valueview.Expert%2CExpertExtender%2CExpertStore%7Cjquery.valueview.experts.CommonsMediaType%2CEmptyValue%2CGeoShape%2CGlobeCoordinateInput%2CMonolingualText%2CQuantityInput%2CStringValue%2CTabularData%2CTimeInput%2CUnDeserializableValue%7Cjquery.wikibase.entityselector%2Cwbtooltip%7Cjquery.wikibase.toolbar.styles&version=1hdw1:0:0
00:04:20.641 10 02 2025 10:14:00.019:DEBUG [Firefox 115.0 (Linux x86_64)]: EXECUTING -> CONNECTED
00:04:20.646 10 02 2025 10:14:00.022:DEBUG [launcher]: CAPTURED -> BEING_KILLED
00:04:20.646 10 02 2025 10:14:00.022:DEBUG [launcher]: BEING_KILLED -> BEING_FORCE_KILLED
00:04:20.646 10 02 2025 10:14:00.022:DEBUG [Firefox 115.0 (Linux x86_64)]: CONNECTED -> DISCONNECTED
00:04:20.646 
00:04:20.646 Finished in 59.326 secs / 57.975 secs @ 10:14:00 GMT+0000 (Coordinated Universal Time)
00:04:20.646 
00:04:20.646 SUMMARY:
00:04:20.647 โœ” 2351 tests completed
00:04:20.647 โ„น 20 tests skipped
00:04:20.647 10 02 2025 10:14:00.023:DEBUG [karma-server]: Run complete, exiting.
00:04:20.647 10 02 2025 10:14:00.023:DEBUG [launcher]: Disconnecting all browsers
00:04:20.647 10 02 2025 10:14:00.023:DEBUG [launcher]: BEING_FORCE_KILLED -> BEING_FORCE_KILLED
00:04:20.647 10 02 2025 10:14:00.024:DEBUG [proxy]: Destroying proxy agents
00:04:20.717 10 02 2025 10:14:00.096:DEBUG [launcher]: Process FirefoxHeadless exited with code null and signal SIGTERM
00:04:20.717 10 02 2025 10:14:00.096:DEBUG [temp-dir]: Cleaning temp dir /tmp/karma-8606850
00:04:20.940 10 02 2025 10:14:00.321:DEBUG [launcher]: Finished all browsers
00:04:20.940 10 02 2025 10:14:00.322:DEBUG [launcher]: BEING_FORCE_KILLED -> FINISHED
00:04:20.940 10 02 2025 10:14:00.322:DEBUG [launcher]: FINISHED -> FINISHED
00:04:20.947 Warning: Task "karma:firefox" failed. Use --force to continue.
00:04:20.947 
00:04:20.948 Aborted due to warnings.

From https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-php74-noselenium/72756/console

Decoding the above URL yields this list of modules:

[
  '@wikimedia/codex',
  'codex-search-styles',
  'codex-styles',
  'color-picker',
  'dataValues',
  'diffMatchPatch',
  'dompurify',
  'mapbox|dataValues.DataValue',
  'TimeValue',
  'values|ext.articleplaceholder.createArticle|ext.centralauth.ForeignApi|ext.checkUser',
  'eventLogging',
  'flow',
  'globalBlocking',
  'thanks',
  'wikimediaEvents|ext.checkUser.clientHints|ext.cite.parsoid.styles|ext.cite.referencePreviews',
  'styles',
  'ux-enhancements',
  'visualEditor|ext.cite.visualEditor.core|ext.discussionTools.init',
  'minervaicons|ext.discussionTools.init.styles|ext.echo.api',
  'dm',
  'init',
  'mobile',
  'secondaryicons',
  'ui|ext.eventLogging.debug|ext.flow.components',
  'templating',
  'ui|ext.flow.jquery.conditionalScroll',
  'findWithParent|ext.flow.visualEditor.icons|ext.graph.render',
  'visualEditor|ext.growthExperiments.DataStore',
  'PostEdit',
  'SuggestedEditSession',
  'icons|ext.growthExperiments.Homepage.SuggestedEdits|ext.guidedTour.lib',
  'styles|ext.guidedTour.lib.internal|ext.kartographer.dialog|ext.popups.main|ext.securepoll.htmlform|ext.templateDataGenerator.data|ext.uls.common',
  'i18n',
  'languagenames',
  'mediawiki',
  'messages',
  'preferences',
  'webfonts|ext.visualEditor.data',
  'icons-licenses',
  'moduleIcons',
  'moduleIndicators|jquery.animateWithEvent',
  'client',
  'highlightText',
  'i18n',
  'inputautoexpand',
  'lengthLimit',
  'makeCollapsible',
  'spinner',
  'tablesorter',
  'textSelection',
  'ui',
  'uls',
  'valueview|jquery.event.special.eachchange|jquery.makeCollapsible.styles|jquery.spinner.styles|jquery.tablesorter.styles|jquery.ui.commonssuggester',
  'languagesuggester',
  'suggester|jquery.uls.data',
  'grid|jquery.util.getDirectionality|jquery.valueview.Expert',
  'ExpertExtender',
  'ExpertStore|jquery.valueview.experts.CommonsMediaType',
  'EmptyValue',
  'GeoShape',
  'GlobeCoordinateInput',
  'MonolingualText',
  'QuantityInput',
  'StringValue',
  'TabularData',
  'TimeInput',
  'UnDeserializableValue|jquery.wikibase.entityselector',
  'wbtooltip|jquery.wikibase.toolbar.styles'
]

Not fully sure what to do with that.

Maybe, assuming this is not caused by VE directly, if we would reduce the overall number of tests, the error would be less likely to happen? In particular, I would like to look into breaking the (very indirect) dependency on Wikibase in CI.

Maybe, assuming this is not caused by VE directly, if we would reduce the overall number of tests, the error would be less likely to happen? In particular, I would like to look into breaking the (very indirect) dependency on Wikibase in CI.

The chain of dependency in CI is: GrowthExperiments -> CirrusSearch -> WikibaseCirrusSearch -> Wikibase.

So, would it be technically possible to not load the WikibaseCirrusSearch dependency (and its dependencies) in GrowthExperiments CI specifically?

Change #1119495 had a related patch set uploaded (by Jforrester; author: Jforrester):

[integration/config@master] Zuul: [mediawiki/extensions/CirrusSearch] Drop WikibaseCirrusSearch dep

https://gerrit.wikimedia.org/r/1119495

Change #1119495 merged by jenkins-bot:

[integration/config@master] Zuul: [mediawiki/extensions/CirrusSearch] Drop WikibaseCirrusSearch dep

https://gerrit.wikimedia.org/r/1119495

Mentioned in SAL (#wikimedia-releng) [2025-02-13T13:57:42Z] <James_F> Zuul: [mediawiki/extensions/CirrusSearch] Drop WikibaseCirrusSearch dep, for T386015

Michael claimed this task.

The problem seems to be fixed, I'm no longer seeing any errors of this kind for our jobs.

For future reference: This change should enable more meaningful stack-traces if this problem ever crops up again: {DNM}: try proper cross-origin request for stacktraces