Page MenuHomePhabricator

Apparent random failing of RevisionSlider qunit tests
Closed, ResolvedPublic

Description

The last successfull run can be found at https://integration.wikimedia.org/ci/job/mwext-qunit-jessie/6407/console from the 30th November.
The first failure run can be found at https://integration.wikimedia.org/ci/job/mwext-qunit-jessie/6711 from the 10 Dec

The failure is confirmed by the job validation patch at https://gerrit.wikimedia.org/r/#/c/326533/ https://integration.wikimedia.org/ci/job/mwext-qunit-jessie/6775/console
Output can be seen below:

20:16:40 Chrome 53.0.2785 (Linux 0.0.0) ext.RevisionSlider.RevisionListView tooltip is composed correctly with en lang FAILED
20:16:40 	Test the page size.
20:16:40 	Expected: true
20:16:40 	Actual: null
20:16:40 	    at Object.<anonymous> (http://localhost:9876/jenkins-mwext-qunit-jessie-6775/load.php?debug=false&lang=en&modules=ext.RevisionSlider.DiffPage%2CHelpDialog%2CPointer%2CPointerLine%2CPointerView%2CRevision%2CRevisionList%2CRevisionListView%2CSlider%2CSliderView%2CdialogImages%2Ctests%7Cjquery.accessKeyLabel%2CautoEllipsis%2CbyteLength%2CbyteLimit%2CcheckboxShiftClick%2Cclient%2Ccolor%2CcolorUtil%2Ccookie%2CgetAttrs%2Chidpi%2ChighlightText%2Clocalize%2CmakeCollapsible%2Cmw-jump%2CmwExtension%2Cplaceholder%2CtabIndex%2Ctablesorter%2CtextSelection%7Cmediawiki.ForeignApi%2CRegExp%2CTitle%2CUri%2Capi%2Ccldr%2Ccookie%2Cexperiments%2CjqueryMsg%2Clanguage%2CmessagePoster%2Cnotify%2Cstorage%2Ctemplate%2Ctoc%2Cuser%2Cutil%2Cviewport%7Cmediawiki.ForeignApi.core%7Cmediawiki.api.category%2Cedit%2Cmessages%2Coptions%2Cparse%2Cupload%2Cuser%2Cwatch%7Cmediawiki.language.data%2Cinit%2Cmonths%7Cmediawiki.libs.pluralruleparser%7Cmediawiki.page.ready%2Cstartup%7Cmediawiki.special.recentchanges%7Cmediawiki.template.mustache%2Cregexp%7Cmoment%2Coojs%2Coojs-ui%2Coojs-ui-core%2Coojs-ui-toolbars%2Coojs-ui-widgets%2Coojs-ui-windows%7Coojs-ui-core.styles%7Coojs-ui.styles.icons%2Cindicators%2Ctextures%7Ctest.mediawiki.qunit.suites%2Ctestrunner%7Ctest.sinonjs%7Cuser.defaults&skin=fallback&version=1be0466:61:373)
20:16:40 	    at runTest (node_modules/qunitjs/qunit/qunit.js:905:28)
20:16:40 	    at Object.run (node_modules/qunitjs/qunit/qunit.js:890:4)
20:16:40 	    at node_modules/qunitjs/qunit/qunit.js:1032:11
20:16:40 	    at process (node_modules/qunitjs/qunit/qunit.js:692:24)
20:16:40 	    at begin (node_modules/qunitjs/qunit/qunit.js:674:2)
20:16:40 	
20:16:40 	Test the change size.
20:16:40 	Expected: true
20:16:40 	Actual: null
20:16:40 	    at Object.<anonymous> (http://localhost:9876/jenkins-mwext-qunit-jessie-6775/load.php?debug=false&lang=en&modules=ext.RevisionSlider.DiffPage%2CHelpDialog%2CPointer%2CPointerLine%2CPointerView%2CRevision%2CRevisionList%2CRevisionListView%2CSlider%2CSliderView%2CdialogImages%2Ctests%7Cjquery.accessKeyLabel%2CautoEllipsis%2CbyteLength%2CbyteLimit%2CcheckboxShiftClick%2Cclient%2Ccolor%2CcolorUtil%2Ccookie%2CgetAttrs%2Chidpi%2ChighlightText%2Clocalize%2CmakeCollapsible%2Cmw-jump%2CmwExtension%2Cplaceholder%2CtabIndex%2Ctablesorter%2CtextSelection%7Cmediawiki.ForeignApi%2CRegExp%2CTitle%2CUri%2Capi%2Ccldr%2Ccookie%2Cexperiments%2CjqueryMsg%2Clanguage%2CmessagePoster%2Cnotify%2Cstorage%2Ctemplate%2Ctoc%2Cuser%2Cutil%2Cviewport%7Cmediawiki.ForeignApi.core%7Cmediawiki.api.category%2Cedit%2Cmessages%2Coptions%2Cparse%2Cupload%2Cuser%2Cwatch%7Cmediawiki.language.data%2Cinit%2Cmonths%7Cmediawiki.libs.pluralruleparser%7Cmediawiki.page.ready%2Cstartup%7Cmediawiki.special.recentchanges%7Cmediawiki.template.mustache%2Cregexp%7Cmoment%2Coojs%2Coojs-ui%2Coojs-ui-core%2Coojs-ui-toolbars%2Coojs-ui-widgets%2Coojs-ui-windows%7Coojs-ui-core.styles%7Coojs-ui.styles.icons%2Cindicators%2Ctextures%7Ctest.mediawiki.qunit.suites%2Ctestrunner%7Ctest.sinonjs%7Cuser.defaults&skin=fallback&version=1be0466:61:433)
20:16:40 	    at runTest (node_modules/qunitjs/qunit/qunit.js:905:28)
20:16:40 	    at Object.run (node_modules/qunitjs/qunit/qunit.js:890:4)
20:16:40 	    at node_modules/qunitjs/qunit/qunit.js:1032:11
20:16:40 	    at process (node_modules/qunitjs/qunit/qunit.js:692:24)
20:16:40 	    at begin (node_modules/qunitjs/qunit/qunit.js:674:2)
20:16:40 	
20:16:40 Chrome 53.0.2785 (Linux 0.0.0): Executed 365 of 378 (1 FAILED) (0 secs / 6.08 secs)
20:16:40 Chrome 53.0.2785 (Linux 0.0.0): Executed 366 of 378 (1 FAILED) (0 secs / 6.081 secs)
20:16:40 Chrome 53.0.2785 (Linux 0.0.0): Executed 367 of 378 (1 FAILED) (0 secs / 6.083 secs)
20:16:40 Chrome 53.0.2785 (Linux 0.0.0): Executed 368 of 378 (1 FAILED) (0 secs / 6.085 secs)
20:16:40 Chrome 53.0.2785 (Linux 0.0.0): Executed 369 of 378 (1 FAILED) (0 secs / 6.086 secs)
20:16:40 Chrome 53.0.2785 (Linux 0.0.0): Executed 370 of 378 (1 FAILED) (0 secs / 6.087 secs)
20:16:40 Chrome 53.0.2785 (Linux 0.0.0): Executed 371 of 378 (1 FAILED) (0 secs / 6.089 secs)
20:16:40 Chrome 53.0.2785 (Linux 0.0.0): Executed 372 of 378 (1 FAILED) (0 secs / 6.091 secs)
20:16:40 Chrome 53.0.2785 (Linux 0.0.0): Executed 373 of 378 (1 FAILED) (0 secs / 6.094 secs)
20:16:40 Chrome 53.0.2785 (Linux 0.0.0) ext.RevisionSlider.RevisionListView big change number is formatted correctly FAILED
20:16:40 	Expected: "+1,000"
20:16:40 	Actual: "+1.000"
20:16:40 	    at Object.<anonymous> (http://localhost:9876/jenkins-mwext-qunit-jessie-6775/load.php?debug=false&lang=en&modules=ext.RevisionSlider.DiffPage%2CHelpDialog%2CPointer%2CPointerLine%2CPointerView%2CRevision%2CRevisionList%2CRevisionListView%2CSlider%2CSliderView%2CdialogImages%2Ctests%7Cjquery.accessKeyLabel%2CautoEllipsis%2CbyteLength%2CbyteLimit%2CcheckboxShiftClick%2Cclient%2Ccolor%2CcolorUtil%2Ccookie%2CgetAttrs%2Chidpi%2ChighlightText%2Clocalize%2CmakeCollapsible%2Cmw-jump%2CmwExtension%2Cplaceholder%2CtabIndex%2Ctablesorter%2CtextSelection%7Cmediawiki.ForeignApi%2CRegExp%2CTitle%2CUri%2Capi%2Ccldr%2Ccookie%2Cexperiments%2CjqueryMsg%2Clanguage%2CmessagePoster%2Cnotify%2Cstorage%2Ctemplate%2Ctoc%2Cuser%2Cutil%2Cviewport%7Cmediawiki.ForeignApi.core%7Cmediawiki.api.category%2Cedit%2Cmessages%2Coptions%2Cparse%2Cupload%2Cuser%2Cwatch%7Cmediawiki.language.data%2Cinit%2Cmonths%7Cmediawiki.libs.pluralruleparser%7Cmediawiki.page.ready%2Cstartup%7Cmediawiki.special.recentchanges%7Cmediawiki.template.mustache%2Cregexp%7Cmoment%2Coojs%2Coojs-ui%2Coojs-ui-core%2Coojs-ui-toolbars%2Coojs-ui-widgets%2Coojs-ui-windows%7Coojs-ui-core.styles%7Coojs-ui.styles.icons%2Cindicators%2Ctextures%7Ctest.mediawiki.qunit.suites%2Ctestrunner%7Ctest.sinonjs%7Cuser.defaults&skin=fallback&version=1be0466:64:480)
20:16:40 	    at runTest (node_modules/qunitjs/qunit/qunit.js:905:28)
20:16:40 	    at Object.run (node_modules/qunitjs/qunit/qunit.js:890:4)
20:16:40 	    at node_modules/qunitjs/qunit/qunit.js:1032:11
20:16:40 	    at process (node_modules/qunitjs/qunit/qunit.js:692:24)
20:16:40 	    at begin (node_modules/qunitjs/qunit/qunit.js:674:2)
20:16:40 	
20:16:40 Chrome 53.0.2785 (Linux 0.0.0): Executed 374 of 378 (2 FAILED) (0 secs / 6.097 secs)
20:16:40 Chrome 53.0.2785 (Linux 0.0.0) ext.RevisionSlider.RevisionListView page size is formatted correctly FAILED
20:16:40 	failed, expected argument to be truthy, was: null
20:16:40 	Expected: true
20:16:40 	Actual: null
20:16:40 	    at Object.<anonymous> (http://localhost:9876/jenkins-mwext-qunit-jessie-6775/load.php?debug=false&lang=en&modules=ext.RevisionSlider.DiffPage%2CHelpDialog%2CPointer%2CPointerLine%2CPointerView%2CRevision%2CRevisionList%2CRevisionListView%2CSlider%2CSliderView%2CdialogImages%2Ctests%7Cjquery.accessKeyLabel%2CautoEllipsis%2CbyteLength%2CbyteLimit%2CcheckboxShiftClick%2Cclient%2Ccolor%2CcolorUtil%2Ccookie%2CgetAttrs%2Chidpi%2ChighlightText%2Clocalize%2CmakeCollapsible%2Cmw-jump%2CmwExtension%2Cplaceholder%2CtabIndex%2Ctablesorter%2CtextSelection%7Cmediawiki.ForeignApi%2CRegExp%2CTitle%2CUri%2Capi%2Ccldr%2Ccookie%2Cexperiments%2CjqueryMsg%2Clanguage%2CmessagePoster%2Cnotify%2Cstorage%2Ctemplate%2Ctoc%2Cuser%2Cutil%2Cviewport%7Cmediawiki.ForeignApi.core%7Cmediawiki.api.category%2Cedit%2Cmessages%2Coptions%2Cparse%2Cupload%2Cuser%2Cwatch%7Cmediawiki.language.data%2Cinit%2Cmonths%7Cmediawiki.libs.pluralruleparser%7Cmediawiki.page.ready%2Cstartup%7Cmediawiki.special.recentchanges%7Cmediawiki.template.mustache%2Cregexp%7Cmoment%2Coojs%2Coojs-ui%2Coojs-ui-core%2Coojs-ui-toolbars%2Coojs-ui-widgets%2Coojs-ui-windows%7Coojs-ui-core.styles%7Coojs-ui.styles.icons%2Cindicators%2Ctextures%7Ctest.mediawiki.qunit.suites%2Ctestrunner%7Ctest.sinonjs%7Cuser.defaults&skin=fallback&version=1be0466:64:903)
20:16:40 	    at runTest (node_modules/qunitjs/qunit/qunit.js:905:28)
20:16:40 	    at Object.run (node_modules/qunitjs/qunit/qunit.js:890:4)
20:16:40 	    at node_modules/qunitjs/qunit/qunit.js:1032:11
20:16:40 	    at process (node_modules/qunitjs/qunit/qunit.js:692:24)
20:16:40 	    at begin (node_modules/qunitjs/qunit/qunit.js:674:2)
20:16:40 	
20:16:40 Chrome 53.0.2785 (Linux 0.0.0): Executed 375 of 378 (3 FAILED) (0 secs / 6.099 secs)
20:16:40 Chrome 53.0.2785 (Linux 0.0.0): Executed 376 of 378 (3 FAILED) (0 secs / 6.099 secs)
20:16:40 Chrome 53.0.2785 (Linux 0.0.0): Executed 377 of 378 (3 FAILED) (0 secs / 6.124 secs)
20:16:40 Chrome 53.0.2785 (Linux 0.0.0): Executed 378 of 378 (3 FAILED) (0 secs / 6.143 secs)
20:16:40 Chrome 53.0.2785 (Linux 0.0.0): Executed 378 of 378 (3 FAILED) (6.386 secs / 6.143 secs)
20:16:40 12 12 2016 20:16:39.337:DEBUG [karma]: Run complete, exiting.
20:16:40 12 12 2016 20:16:39.339:DEBUG [launcher]: Disconnecting all browsers
20:16:40 12 12 2016 20:16:39.863:DEBUG [launcher]: Process Chrome exited with code 0
20:16:40 12 12 2016 20:16:39.864:DEBUG [temp-dir]: Cleaning temp dir /home/jenkins/tmpfs/jenkins-0/karma-26044742
20:16:40 12 12 2016 20:16:39.921:DEBUG [launcher]: Finished all browsers
20:16:40 Warning: Task "karma:main" failed. Use --force to continue.
20:16:40 
20:16:40 Aborted due to warnings.

It looks like these can be reproduced locally

Event Timeline

Restricted Application added a project: TCB-Team. · View Herald TranscriptDec 13 2016, 8:34 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Addshore updated the task description. (Show Details)Dec 13 2016, 8:42 PM

@Addshore can you describe a bit more under which circumstances this was reproducible locally?

FWIW I also finally managed to reproduce the failures, somehow. So I have a setup with only core and RevisionSlider extension enabled. When I run all Qunit tests, those 3 RS tests fail. When I rerun all tests in most cases those tests pass (although I didn't verify this scientifically enough to be able to say they only fail on the first test run). Also, when I only run test for the module in which failing test are, they have been passing in all my attempts.
So there might either be some race condition involved, or maybe some of core tests dealing with language setting is affecting RS test in a way that they fail. Will be trying to figure out more what's going on there.
Maybe we should try to make RS tests dealing with user language more reliable. What's done over there seems a bit hacky, after all.

Change 327166 had a related patch set uploaded (by WMDE-leszek):
Reset custom separator transform tables in number formatting tests

https://gerrit.wikimedia.org/r/327166

Change 327166 merged by jenkins-bot:
Reset custom separator transform tables in number formatting tests

https://gerrit.wikimedia.org/r/327166

Tobi_WMDE_SW closed this task as Resolved.Dec 14 2016, 12:33 PM
Tobi_WMDE_SW claimed this task.
Tobi_WMDE_SW moved this task from Watching/Blocked/External to Demoed on the WMDE-QWERTY-Team board.

Due to mediawiki/core change https://gerrit.wikimedia.org/r/#/c/324387/

Congratulations to have reproduced and fixed it !

Change 327499 had a related patch set uploaded (by Addshore):
Reset custom separator transform tables in number formatting tests

https://gerrit.wikimedia.org/r/327499

Change 327499 merged by jenkins-bot:
Reset custom separator transform tables in number formatting tests

https://gerrit.wikimedia.org/r/327499

Tobi_WMDE_SW moved this task from Incoming to Done on the Revision-Slider board.Dec 16 2016, 10:34 AM
Tobi_WMDE_SW moved this task from Demoed to Done on the WMDE-QWERTY-Team board.Dec 16 2016, 2:52 PM
Tobi_WMDE_SW moved this task from Done to Demoed on the WMDE-QWERTY-Team board.Dec 20 2016, 12:16 PM