Page MenuHomePhabricator

MobileFrontend: mobileView references gateway qunit test is flaky
Closed, ResolvedPublic

Description

A MobileFrontend unit test has apparently been failing randomly for the past two weeks-ish.

The following example on the Echo extension:

ULS:

17:22:32 SUMMARY:
17:22:32 ✔ 1257 tests completed
17:22:32 ✖ 1 test failed
17:22:32 
17:22:32 FAILED TESTS:
17:22:32   MobileFrontend: mobileView references gateway
17:22:32     ✖ checking good reference (subsequent calls)
17:22:32       Chrome 57.0.2987 (Linux 0.0.0)
17:22:32     Expected at least one assertion, but none were run - call expect(0) to accept zero assertions.
17:22:32         at http://localhost:9876/jenkins-mediawiki-extensions-qunit-jessie-44610/load.php?debug=false&lang=en&modules=mediawiki.special.recentchanges%7Cmediawiki.storage%2Ctemplate%2Ctoc%2Cui%2Cuser%2Cutil%2Cviewport%2Cwidgets%7Cmediawiki.template.handlebars%2Chogan%2Cmustache%2Cregexp%7Cmediawiki.ui.anchor%2Cbutton%2Cicon%7Cmediawiki.widgets.CategoryMultiselectWidget%2CDateInputWidget%2CMediaSearch%2CStashedFileWidget%2CUserInputWidget%2Cstyles%7Cmediawiki.widgets.DateInputWidget.styles%7Cmobile.ajax.styles%7Cmobile.editor.api%2Ccommon%2Coverlay%7Cmobile.foreignApi%2Cgallery%2CinfiniteScroll%2CmediaViewer%2CmessageBox%2Cnearby%2Creferences%2Csearch%2Cstartup%2Ctoggle%2Cwatchlist%2Cwatchstar%7Cmobile.languages.structured%7Cmobile.loggingSchemas.edit%2CmobileWebSearch%7Cmobile.messageBox.styles%7Cmobile.overlay.images%7Cmobile.pagelist.scripts%2Cstyles%7Cmobile.pagesummary.styles%7Cmobile.references.gateway%2Cimages%7Cmobile.search.api%2Cimages%2Cutil%7Cmobile.special.nearby.styles%7Cmobile.startup.images%7Cmobile.talk.overlays%7Cmoment%2Coojs%2Coojs-router%2Coojs-ui%2Coojs-ui-core%2Coojs-ui-toolbars%2Coojs-ui-widgets%2Coojs-ui-windows%2Cpapaparse%2Crangefix%2Cspark-md5%2CtreeDiffer%2Cunicodejs%7Cmw.config.values.wbDataTypes%2CwbRepo%2CwbSiteDetails%7Coojs-ui-core.styles%7Coojs-ui.styles.icons-accessibility%2Cicons-alerts%2Cicons-content%2Cicons-editing-advanced%2Cicons-editing-core%2Cicons-editing-list%2Cicons-editing-styling%2Cicons-interactions%2Cicons-layout%2Cicons-media%2Cicons-moderation%2Cicons-movement%2Cicons-user%2Cicons-wikimedia%2Cindicators%2Ctextures%7Cschema.GuidedTourButtonClick%2CGuidedTourExited%2CGuidedTourExternalLinkActivation%2CGuidedTourGuiderHidden%2CGuidedTourGuiderImpression%2CGuidedTourInternalLinkActivation%2CMobileWebSearch%2CNavigationTiming%2CSaveTiming%7Csocket.io%7Ctest.mediawiki.qunit.suites%2Ctestrunner%7Ctest.sinonjs%7Ctests.ext.thanks.mobilediff%7Ctests.mobilefrontend%2Czero%7Cuser.defaults%7Cutil.CombiningMessageProvider%2CExtendable%2CHashMessageProvider%2CMessageProvider%2CNotifier&skin=fallback&version=0pd0jnf:1526:836
17:22:32         at mw.loader.implement.section.hogan (http://localhost:9876/jenkins-mediawiki-extensions-qunit-jessie-44610/load.php?debug=false&lang=en&modules=mediawiki.special.recentchanges%7Cmediawiki.storage%2Ctemplate%2Ctoc%2Cui%2Cuser%2Cutil%2Cviewport%2Cwidgets%7Cmediawiki.template.handlebars%2Chogan%2Cmustache%2Cregexp%7Cmediawiki.ui.anchor%2Cbutton%2Cicon%7Cmediawiki.widgets.CategoryMultiselectWidget%2CDateInputWidget%2CMediaSearch%2CStashedFileWidget%2CUserInputWidget%2Cstyles%7Cmediawiki.widgets.DateInputWidget.styles%7Cmobile.ajax.styles%7Cmobile.editor.api%2Ccommon%2Coverlay%7Cmobile.foreignApi%2Cgallery%2CinfiniteScroll%2CmediaViewer%2CmessageBox%2Cnearby%2Creferences%2Csearch%2Cstartup%2Ctoggle%2Cwatchlist%2Cwatchstar%7Cmobile.languages.structured%7Cmobile.loggingSchemas.edit%2CmobileWebSearch%7Cmobile.messageBox.styles%7Cmobile.overlay.images%7Cmobile.pagelist.scripts%2Cstyles%7Cmobile.pagesummary.styles%7Cmobile.references.gateway%2Cimages%7Cmobile.search.api%2Cimages%2Cutil%7Cmobile.special.nearby.styles%7Cmobile.startup.images%7Cmobile.talk.overlays%7Cmoment%2Coojs%2Coojs-router%2Coojs-ui%2Coojs-ui-core%2Coojs-ui-toolbars%2Coojs-ui-widgets%2Coojs-ui-windows%2Cpapaparse%2Crangefix%2Cspark-md5%2CtreeDiffer%2Cunicodejs%7Cmw.config.values.wbDataTypes%2CwbRepo%2CwbSiteDetails%7Coojs-ui-core.styles%7Coojs-ui.styles.icons-accessibility%2Cicons-alerts%2Cicons-content%2Cicons-editing-advanced%2Cicons-editing-core%2Cicons-editing-list%2Cicons-editing-styling%2Cicons-interactions%2Cicons-layout%2Cicons-media%2Cicons-moderation%2Cicons-movement%2Cicons-user%2Cicons-wikimedia%2Cindicators%2Ctextures%7Cschema.GuidedTourButtonClick%2CGuidedTourExited%2CGuidedTourExternalLinkActivation%2CGuidedTourGuiderHidden%2CGuidedTourGuiderImpression%2CGuidedTourInternalLinkActivation%2CMobileWebSearch%2CNavigationTiming%2CSaveTiming%7Csocket.io%7Ctest.mediawiki.qunit.suites%2Ctestrunner%7Ctest.sinonjs%7Ctests.ext.thanks.mobilediff%7Ctests.mobilefrontend%2Czero%7Cuser.defaults%7Cutil.CombiningMessageProvider%2CExtendable%2CHashMessageProvider%2CMessageProvider%2CNotifier&skin=fallback&version=0pd0jnf:1528:128)
17:22:32         at runScript (http://localhost:9876/jenkins-mediawiki-extensions-qunit-jessie-44610/load.php?debug=false&lang=en&modules=jquery%2Cmediawiki&only=scripts&skin=fallback&version=05gybsn:163:129)
17:22:32         at checkCssHandles (http://localhost:9876/jenkins-mediawiki-extensions-qunit-jessie-44610/load.php?debug=false&lang=en&modules=jquery%2Cmediawiki&only=scripts&skin=fallback&version=05gybsn:163:785)
17:22:32         at execute (http://localhost:9876/jenkins-mediawiki-extensions-qunit-jessie-44610/load.php?debug=false&lang=en&modules=jquery%2Cmediawiki&only=scripts&skin=fallback&version=05gybsn:164:496)
17:22:32         at handlePending (http://localhost:9876/jenkins-mediawiki-extensions-qunit-jessie-44610/load.php?debug=false&lang=en&modules=jquery%2Cmediawiki&only=scripts&skin=fallback&version=05gybsn:160:80)
17:22:32         at markModuleReady (http://localhost:9876/jenkins-mediawiki-extensions-qunit-jessie-44610/load.php?debug=false&lang=en&modules=jquery%2Cmediawiki&only=scripts&skin=fallback&version=05gybsn:162:816)

Developer notes

The issue (to my knowledge) hasn't shown up in the MobileFrontend or Minerva repo. We shouldn't rule out the possibility that the issue is actually not with this test but some other test in another extension. The more examples of failing builds that we have, the better.

Details

Related Gerrit Patches:
mediawiki/extensions/MobileFrontend : masterReturn a Deferred that always resolves
mediawiki/extensions/MobileFrontend : masterAdd debugging information to flakey test

Event Timeline

Legoktm created this task.Nov 7 2017, 5:39 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 7 2017, 5:39 PM

@Niedzielski I don't see how this is related to T162256 - this is about qunit tests being flaky, not browser tests.

Jdlrobson updated the task description. (Show Details)Nov 9 2017, 7:32 PM
Jdlrobson moved this task from Incoming to Needs Prioritization on the Readers-Web-Backlog board.
Jdlrobson added a subscriber: Jdlrobson.

Did you just see this on Echo or were other extensions involved? I've not personally seen this issue, so I'm keen to rule out whether any other extensions are interfering with this test.

Restricted Application added a project: Collaboration-Team-Triage. · View Herald TranscriptNov 13 2017, 5:13 PM

The test in question looks like this:

	QUnit.test( 'checking good reference (subsequent calls)', function ( assert ) {
		var page = this.page,
			referencesGateway = this.referencesGateway;

		return referencesGateway.getReference( '#cite_note-1', page ).always( function () {
			return referencesGateway.getReference( '#cite_note-2', page ).done( function ( ref ) {
				assert.strictEqual( ref.text, 'real lazy 2' );
			} );
		} );
	} );

The error I'm seeing is Expected at least one assertion, but none were run - call expect(0) to accept zero assertions. which is odd as it sounds like the call to getReference is failing.

This will only fail if an exception is thrown or the promise is rejected.
I'm a little confused though, as if that was the case I'd expect `Promise rejected during checking good reference" not the message "Expected at least one assertion"

That error message should only happen if the promise resolves but the assertion fails to happen. Given the assertion looks like

assert.strictEqual( ref.text, 'real lazy' );

it seems that could only happen if ref is undefined.

So this makes me wonder why ref is undefined.
getReference's logic is quite simple.

		getReference: function ( id, page ) {
			var self = this;

			return this.getReferencesLists( page ).then( function ( sections ) {
				var $container = $( '<div>' );

				Object.keys( sections ).forEach( function ( sectionId ) {
					$container.append( sections[ sectionId ] );
				} );

				return self.getReferenceFromContainer( id, $container );
			} );
		}

getReferenceFromContainer is only resolved in one place:

result = $.Deferred(),
...
			if ( $el.length ) {
				result.resolve( { text: $el.html() } );
			} else {
				result.reject( ReferencesGateway.ERROR_NOT_EXIST );
			}

That would only resolve with undefined if something was hijacking $.Deferred to resolve with undefined which seems unlikely as the other tests are passing OR if getReference was being stubbed somewhere - but I see no evidence of that either.

Thus I'm a little stumped with what's going on here.

Are people still running into this problem?

Jdlrobson moved this task from Inbox to Next up on the User-Jdlrobson board.Nov 29 2017, 8:21 PM

Change 401821 had a related patch set uploaded (by Jdlrobson; owner: Jdlrobson):
[mediawiki/extensions/MobileFrontend@master] Add debugging information to flakey test

https://gerrit.wikimedia.org/r/401821

Jdlrobson added a comment.EditedJan 4 2018, 12:11 AM

Seen this myself but still can't replicate it. I think the above should help us debug this better (getReference should only fail if there's a JS error of some sort) and get to the bottom of this if somebody can merge!

Change 401821 merged by jenkins-bot:
[mediawiki/extensions/MobileFrontend@master] Add debugging information to flakey test

https://gerrit.wikimedia.org/r/401821

Jdlrobson changed the task status from Open to Stalled.Jan 19 2018, 1:18 AM

Waiting on a new bug report - once we see this happen again we should get more insight into the problem...

Is this still a problem?

Is this still a problem?

Same question from me, especially with regards to ULS.

Shall we set a countdown clock and close it? :)

Jdlrobson moved this task from Next up to Blocked on the User-Jdlrobson board.Mar 6 2018, 5:54 PM
Reedy added a subscriber: Reedy.Mar 16 2018, 12:52 AM

Yes, it's still a problem

https://integration.wikimedia.org/ci/job/mediawiki-extensions-qunit-jessie/56902/console

00:45:41 FAILED TESTS:
00:45:41   MobileFrontend: mobileView references gateway
00:45:41     ✖ checking good reference (subsequent calls)
00:45:41       Chrome 57.0.2987 (Linux 0.0.0)
00:45:41     Expected at least one assertion, but none were run - call expect(0) to accept zero assertions.
00:45:41         at http://localhost:9876/jenkins-mediawiki-extensions-qunit-jessie-56902/load.php?debug=false&lang=en&modules=mediawiki.page.startup%7Cmediawiki.rcfilters.filters.dm%7Cmediawiki.router%2Cstorage%2Ctemplate%2Ctoc%2Cui%2Cuser%2Cutil%2Cviewport%2CvisibleTimeout%2Cwidgets%7Cmediawiki.skinning.content.parsoid%7Cmediawiki.special.recentchanges%7Cmediawiki.template.handlebars%2Chogan%2Cmustache%2Cregexp%7Cmediawiki.ui.anchor%2Cbutton%2Cicon%7Cmediawiki.widgets.CategoryMultiselectWidget%2CDateInputWidget%2CMediaSearch%2CStashedFileWidget%2CUserInputWidget%2Cstyles%7Cmediawiki.widgets.DateInputWidget.styles%7Cmobile.ajax.styles%7Cmobile.editor.api%2Ccommon%2Coverlay%7Cmobile.foreignApi%2Cgallery%2CinfiniteScroll%2CmediaViewer%2CmessageBox%2Cnearby%2Creferences%2Csearch%2Cstartup%2Ctoggle%2Cwatchlist%2Cwatchstar%7Cmobile.languages.structured%7Cmobile.loggingSchemas.edit%2CmobileWebSearch%7Cmobile.messageBox.styles%7Cmobile.nearby.images%7Cmobile.pagelist.scripts%2Cstyles%7Cmobile.pagesummary.styles%7Cmobile.references.gateway%2Cimages%7Cmobile.search.api%2Cimages%2Cutil%7Cmobile.special.nearby.styles%7Cmobile.startup.images%7Cmobile.startup.images.variants%7Cmobile.talk.overlays%7Cmoment%2Coojs%2Coojs-router%2Coojs-ui%2Coojs-ui-core%2Coojs-ui-toolbars%2Coojs-ui-widgets%2Coojs-ui-windows%2Cpapaparse%2Crangefix%2Cspark-md5%7Cmw.config.values.wbDataTypes%2CwbRepo%2CwbSiteDetails%7Coojs-ui-core.styles%7Coojs-ui.styles.icons-accessibility%2Cicons-alerts%2Cicons-content%2Cicons-editing-advanced%2Cicons-editing-core%2Cicons-editing-list%2Cicons-editing-styling%2Cicons-interactions%2Cicons-layout%2Cicons-media%2Cicons-moderation%2Cicons-movement%2Cicons-user%2Cicons-wikimedia%2Cindicators%2Ctextures%7Cschema.GuidedTourButtonClick%2CGuidedTourExited%2CGuidedTourExternalLinkActivation%2CGuidedTourGuiderHidden%2CGuidedTourGuiderImpression%2CGuidedTourInternalLinkActivation%2CMobileWebSearch%2CNavigationTiming%2CSaveTiming%7Csocket.io%7Ctest.mediawiki.qunit.suites%2Ctestrunner%7Ctest.sinonjs%7Ctests.ext.thanks.mobilediff%7Ctests.mobilefrontend%2Czero&skin=fallback&version=0ol4ty9:1581:185
00:45:41         at mw.loader.implement.section.hogan (http://localhost:9876/jenkins-mediawiki-extensions-qunit-jessie-56902/load.php?debug=false&lang=en&modules=mediawiki.page.startup%7Cmediawiki.rcfilters.filters.dm%7Cmediawiki.router%2Cstorage%2Ctemplate%2Ctoc%2Cui%2Cuser%2Cutil%2Cviewport%2CvisibleTimeout%2Cwidgets%7Cmediawiki.skinning.content.parsoid%7Cmediawiki.special.recentchanges%7Cmediawiki.template.handlebars%2Chogan%2Cmustache%2Cregexp%7Cmediawiki.ui.anchor%2Cbutton%2Cicon%7Cmediawiki.widgets.CategoryMultiselectWidget%2CDateInputWidget%2CMediaSearch%2CStashedFileWidget%2CUserInputWidget%2Cstyles%7Cmediawiki.widgets.DateInputWidget.styles%7Cmobile.ajax.styles%7Cmobile.editor.api%2Ccommon%2Coverlay%7Cmobile.foreignApi%2Cgallery%2CinfiniteScroll%2CmediaViewer%2CmessageBox%2Cnearby%2Creferences%2Csearch%2Cstartup%2Ctoggle%2Cwatchlist%2Cwatchstar%7Cmobile.languages.structured%7Cmobile.loggingSchemas.edit%2CmobileWebSearch%7Cmobile.messageBox.styles%7Cmobile.nearby.images%7Cmobile.pagelist.scripts%2Cstyles%7Cmobile.pagesummary.styles%7Cmobile.references.gateway%2Cimages%7Cmobile.search.api%2Cimages%2Cutil%7Cmobile.special.nearby.styles%7Cmobile.startup.images%7Cmobile.startup.images.variants%7Cmobile.talk.overlays%7Cmoment%2Coojs%2Coojs-router%2Coojs-ui%2Coojs-ui-core%2Coojs-ui-toolbars%2Coojs-ui-widgets%2Coojs-ui-windows%2Cpapaparse%2Crangefix%2Cspark-md5%7Cmw.config.values.wbDataTypes%2CwbRepo%2CwbSiteDetails%7Coojs-ui-core.styles%7Coojs-ui.styles.icons-accessibility%2Cicons-alerts%2Cicons-content%2Cicons-editing-advanced%2Cicons-editing-core%2Cicons-editing-list%2Cicons-editing-styling%2Cicons-interactions%2Cicons-layout%2Cicons-media%2Cicons-moderation%2Cicons-movement%2Cicons-user%2Cicons-wikimedia%2Cindicators%2Ctextures%7Cschema.GuidedTourButtonClick%2CGuidedTourExited%2CGuidedTourExternalLinkActivation%2CGuidedTourGuiderHidden%2CGuidedTourGuiderImpression%2CGuidedTourInternalLinkActivation%2CMobileWebSearch%2CNavigationTiming%2CSaveTiming%7Csocket.io%7Ctest.mediawiki.qunit.suites%2Ctestrunner%7Ctest.sinonjs%7Ctests.ext.thanks.mobilediff%7Ctests.mobilefrontend%2Czero&skin=fallback&version=0ol4ty9:1582:523)
00:45:41         at runScript (http://localhost:9876/jenkins-mediawiki-extensions-qunit-jessie-56902/load.php?debug=false&lang=en&modules=jquery%2Cmediawiki&only=scripts&skin=fallback&version=1wn9jw1:163:129)
00:45:41         at checkCssHandles (http://localhost:9876/jenkins-mediawiki-extensions-qunit-jessie-56902/load.php?debug=false&lang=en&modules=jquery%2Cmediawiki&only=scripts&skin=fallback&version=1wn9jw1:163:785)
00:45:41         at execute (http://localhost:9876/jenkins-mediawiki-extensions-qunit-jessie-56902/load.php?debug=false&lang=en&modules=jquery%2Cmediawiki&only=scripts&skin=fallback&version=1wn9jw1:164:496)
00:45:41         at handlePending (http://localhost:9876/jenkins-mediawiki-extensions-qunit-jessie-56902/load.php?debug=false&lang=en&modules=jquery%2Cmediawiki&only=scripts&skin=fallback&version=1wn9jw1:160:80)
00:45:41         at markModuleReady (http://localhost:9876/jenkins-mediawiki-extensions-qunit-jessie-56902/load.php?debug=false&lang=en&modules=jquery%2Cmediawiki&only=scripts&skin=fallback&version=1wn9jw1:162:816)
00:45:41     
00:45:41

Change 420068 had a related patch set uploaded (by Jdlrobson; owner: Jdlrobson):
[mediawiki/extensions/MobileFrontend@master] Return a Deferred that always resolves

https://gerrit.wikimedia.org/r/420068

Krinkle triaged this task as Unbreak Now! priority.EditedMar 20 2018, 2:21 AM
Krinkle added a subscriber: Krinkle.

Still breaking mediawiki/core commits, e.g. https://gerrit.wikimedia.org/r/#/c/420617/

FAILED TESTS:
  MobileFrontend: mobileView references gateway
    ✖ checking good reference (subsequent calls)
      Chrome 57.0.2987 (Linux 0.0.0)
    Expected at least one assertion, but none were run - call expect(0) to accept zero assertions.

Please address soon, or simply disable the test for the time being.

Restricted Application added subscribers: Liuxinyu970226, TerraCodes. · View Herald TranscriptMar 20 2018, 2:21 AM
Krinkle changed the task status from Stalled to Open.Mar 20 2018, 2:21 AM
Jdlrobson added a subscriber: ovasileva.

@ovasileva this has been marked unbreak now. We're impacting other teams so should fix this ASAP.

Restricted Application edited projects, added Readers-Web-Backlog; removed Readers-Web-Backlog (Tracking). · View Herald TranscriptMar 20 2018, 3:52 PM

Sounds good. Let's do it.

Change 420068 merged by jenkins-bot:
[mediawiki/extensions/MobileFrontend@master] Return a Deferred that always resolves

https://gerrit.wikimedia.org/r/420068

Jdlrobson lowered the priority of this task from Unbreak Now! to High.Mar 20 2018, 11:00 PM

Leaving as high until I get confirmation that the problem is resolved... (or not)

I'm gonna assume this is fixed. If not, please let me know if it isn't with more examples!

Jdlrobson closed this task as Resolved.Mar 23 2018, 7:00 PM

(Please reopen and bump back to unbreak now if it's not)

mmodell changed the subtype of this task from "Task" to "Production Error".Aug 28 2019, 11:09 PM