Find out why Lexeme:Forms test is flaky in CI and fix
Closed, InvalidPublic
Actions

Assigned To

None

Authored By

	Michael
	Jul 26 2019, 10:45 AM

Description

The test Lexeme:Forms FormId counter is not decremented when old revision is restored fails occasionally in CI with the error message: element (".lemma-widget_edit") still not visible after 10000ms

Example failure: https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/Wikibase/+/527549/

"stack trace" from the selenium node job log (https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-php70-docker/26829/console):

11:21:27 1) Lexeme:Forms FormId counter is not decremented when old revision is restored:
11:21:27 element (".lemma-widget_edit") still not visible after 10000ms

11:21:27 Error: element (".lemma-widget_edit") still not visible after 10000ms
11:21:27     at elements(".lemma-widget_edit") - isVisible.js:54:17
11:21:27     at isVisible(".lemma-widget_edit") - waitForVisible.js:73:22

Looking at video recording of the failed test runs it looks like the UI JavaScript is not loaded (UI is not fully "initialized").

Acceptance criteria:

The reason of unrelated failure of the test is documented here in the task as a comment
The reason of failures has been removed/worked around

Currently on investigation

Details

Subject	Repo	Branch	Lines +/-
selenium: consistently use waitForVisible on element	mediawiki/extensions/WikibaseLexeme	master	+8 -3
selenium: log browser logs to node console on failure	mediawiki/extensions/WikibaseLexeme	master	+5 -0
Disable CirrusSearch in browser tests	mediawiki/extensions/Wikibase	master	+6 -0

Customize query in gerrit

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Duplicate	Goal	None	T234012 Eliminate Wikidata Browser Tests Flakiness
		Invalid		None	T229099 Find out why Lexeme:Forms test is flaky in CI and fix

Event Timeline

Michael created this task.Jul 26 2019, 10:45 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 26 2019, 10:45 AM

Michael triaged this task as High priority.Aug 1 2019, 2:16 PM

WMDE-leszek updated the task description. (Show Details)Aug 5 2019, 2:47 PM

WMDE-leszek updated the task description. (Show Details)Aug 5 2019, 2:49 PM

WMDE-leszek updated the task description. (Show Details)Aug 5 2019, 3:09 PM

This needs a long term solution.

@noarave , @Ladsgroup and @alaa_wmde will investigate

darthmon_wmde updated the task description. (Show Details)Aug 6 2019, 12:28 PM

darthmon_wmde moved this task from Ready to estimate to Ready to pick up on the Wikidata-Campsite board.

Lucas_Werkmeister_WMDE subscribed.Aug 7 2019, 12:06 PM

Looking at the video recording of the failing test, e.g. https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-php70-docker/26829/artifact/log/Lexeme%253ALemma-can-not-save-lemmas-with-redundant-languages.mp4 it looks like if the JavaScript wasn't loaded. It might be worthwhile to look at browser console log, if there were any (Resource Loader?) errors popping up.
It seems there is no structured way to log client/browser errors to some MW's logging facility, so random brutal way to check log could be to check in the selenium test code if there are errors in the browser reported, and if so, dump them to test logs, i.e. something similar to what Ruby tests do: https://github.com/wikimedia/mediawiki-extensions-Wikibase/blob/master/tests/browser/features/support/env.rb#L46-L58

Change 529042 had a related patch set uploaded (by Pablo Grass (WMDE); owner: Pablo Grass (WMDE)):
[mediawiki/extensions/WikibaseLexeme@master] selenium: log browser logs to node console on failure

https://gerrit.wikimedia.org/r/529042

gerritbot added a project: Patch-For-Review.Aug 8 2019, 8:30 AM

CI now logs the browser console to node console on failure (same moment in time when taking the screenshots) - not necessarily conclusive, though.
e.g. https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-hhvm-docker/64618/console

Change 529054 had a related patch set uploaded (by Pablo Grass (WMDE); owner: Pablo Grass (WMDE)):
[mediawiki/extensions/WikibaseLexeme@master] selenium: consistently use waitForVisible on element

https://gerrit.wikimedia.org/r/529054

• Pablo-WMDE claimed this task.Aug 8 2019, 9:59 AM

• Pablo-WMDE edited projects, added Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)); removed Wikidata-Campsite.

• Pablo-WMDE moved this task from To Do (prioritised from top to bottom) to Doing on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.

Jakob_WMDE claimed this task.Aug 28 2019, 11:00 AM

Change 533956 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[mediawiki/extensions/Wikibase@master] Disable CirrusSearch in browser tests

https://gerrit.wikimedia.org/r/533956

Jakob_WMDE removed Jakob_WMDE as the assignee of this task.Sep 3 2019, 10:28 AM

Jakob_WMDE moved this task from Doing to To Do (prioritised from top to bottom) on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.

Jakob_WMDE subscribed.

Ladsgroup moved this task from To Do (prioritised from top to bottom) to Peer Review on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.Sep 3 2019, 1:03 PM

Ladsgroup mentioned this in T231685: Flaky test: Lexeme:Forms can cancel form addition.Sep 3 2019, 1:22 PM

Change 533956 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Disable CirrusSearch in browser tests

https://gerrit.wikimedia.org/r/533956

The patch should have fixed the issue hopefully. Feel free to bring it back if flaky tests keep happening

Ladsgroup claimed this task.Sep 3 2019, 4:25 PM

Restricted Application added a project: User-Ladsgroup. · View Herald TranscriptSep 3 2019, 4:25 PM

ReleaseTaggerBot added a project: MW-1.34-notes (1.34.0-wmf.22; 2019-09-10).Sep 3 2019, 5:00 PM

Acceptance criteria listed in the task include "The reason of unrelated failure of the test is documented here in the task as a comment".
I don't think the explanation why/how the change made in https://gerrit.wikimedia.org/r/533956 was related to the issues noticed with browser tests.
Would you mind @Ladsgroup writing down a couple of sentences here for the future generations?

• alaa_wmde added a parent task: T234012: Eliminate Wikidata Browser Tests Flakiness.Sep 27 2019, 7:35 AM

• alaa_wmde edited projects, added Wikidata-Campsite; removed Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)).Oct 1 2019, 2:01 PM

• alaa_wmde removed a project: Wikidata-Campsite.Oct 2 2019, 10:33 AM

Ladsgroup removed Ladsgroup as the assignee of this task.Oct 23 2019, 10:20 AM

Ladsgroup added a project: Wikidata Lexicographical data.

Change 529042 abandoned by Pablo Grass (WMDE):
selenium: log browser logs to node console on failure

https://gerrit.wikimedia.org/r/529042

Change 529054 abandoned by Pablo Grass (WMDE):
selenium: consistently use waitForVisible on element

https://gerrit.wikimedia.org/r/529054

Seemingly this is still a flakey browser test per monitoring at T277205

Addshore moved this task from Inbox to Research on the [DEPRECATED] wdwb-tech board.Apr 14 2021, 4:08 PM

Addshore moved this task from Research to Investigate & Discuss on the [DEPRECATED] wdwb-tech board.Jul 16 2021, 8:38 AM

Per T277205 it doesnt look like this has failed in the past 3-4 weeks

	F34392067: image.png
	Apr 14 2021, 4:07 PM

Find out why Lexeme:Forms test is flaky in CI and fixClosed, InvalidPublicActions

Description

Details

Related ObjectsSearch...

Event Timeline

Find out why Lexeme:Forms test is flaky in CI and fix
Closed, InvalidPublic
Actions

Related Objects
Search...