Page MenuHomePhabricator

Make WikibaseLexeme browser tests less flaky
Open, NormalPublic

Description

Flakiness in WikibaseLexeme lately seems to have increased, resulting in delayed CI delivery due to failures that call to running pipelines again and again.

There could be a part of those tests that is too fragile to the environment or external dependencies, and can be made less fragile. It could be something completely different.

We need to investigate, understand why those tests tend to falsely fail that often and patch up some of the issues to decrease their flakiness.


Copied from T224596: Wikibase browser tests blocking merge of unrelated patches:

Trying to merge gerrit:512926 but Wikibase browser tests keep failing.

Output from the last failures:

10:24:10 1) Lexeme:Senses Adding Sense:
10:24:10 element (".wikibase-lexeme-sense-gloss > .wikibase-lexeme-sense-gloss-value-cell > span") still not existing after 10000ms
10:24:10 running chrome
10:24:10 Error: element (".wikibase-lexeme-sense-gloss > .wikibase-lexeme-sense-gloss-value-cell > span") still not existing after 10000ms
10:24:10     at SensePage.getNthSenseData (extensions/WikibaseLexeme/tests/selenium/pageobjects/sense.page.js:91:66)
10:24:10     at Context.it (extensions/WikibaseLexeme/tests/selenium/specs/sense.js:30:25)
10:24:10     at Promise.F (node_modules/core-js/library/modules/_export.js:36:28)
10:24:10     at elements(".wikibase-lexeme-sense-gloss > .wikibase-lexeme-sense-gloss-value-cell > span") - isExisting.js:46:17
10:24:10     at isExisting(".wikibase-lexeme-sense-gloss > .wikibase-lexeme-sense-gloss-value-cell > span") - waitForExist.js:67:22
10:24:10 
10:24:10 2) Lexeme:Senses Added Sense has statement:
10:24:10 An element could not be located on the page using the given search parameters (".wikibase-lexeme-sense-statements h2 > span").
10:24:10 running chrome
10:24:10 Error: An element could not be located on the page using the given search parameters (".wikibase-lexeme-sense-statements h2 > span").
10:24:10     at Context.it (extensions/WikibaseLexeme/tests/selenium/specs/sense.js:45:34)
10:24:10     at Promise.F (node_modules/core-js/library/modules/_export.js:36:28)
10:24:10     at getText() - at SensePage.get senseStatements [as senseStatements] (extensions/WikibaseLexeme/tests/selenium/pageobjects/sense.page.js:46:61)
10:24:10 
10:24:10 3) Lexeme:Senses Anchor exists and is equal to Sense ID:
10:24:10 undefined === ''
10:24:10 running chrome
10:24:10 AssertionError: undefined === ''
10:24:10     at Context.it (/workspace/src/extensions/WikibaseLexeme/tests/selenium/specs/sense.js:55:10)
10:24:10     at Promise.F (/workspace/src/node_modules/core-js/library/modules/_export.js:36:28)

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 29 2019, 8:53 AM

Change 513137 had a related patch set uploaded (by Kosta Harlan; owner: Kosta Harlan):
[mediawiki/extensions/WikibaseLexeme@master] Disable flaky tests

https://gerrit.wikimedia.org/r/513137

SBisson triaged this task as Unbreak Now! priority.May 29 2019, 3:03 PM

This is currently blocking merges in the GrowthExperiments extension.

Restricted Application added a subscriber: Liuxinyu970226. · View Herald TranscriptMay 29 2019, 3:03 PM

Since this also seems to affect non-Wikibase extensions (e. g. the GrowthExperiments patch linked in the task description), that makes it a ci-test-error, which are usually UBN! as far as I understand.

Change 513137 merged by jenkins-bot:
[mediawiki/extensions/WikibaseLexeme@master] Disable flaky tests

https://gerrit.wikimedia.org/r/513137

That seems to be a separate problem, which has now been filed as T224607.

Lucas_Werkmeister_WMDE lowered the priority of this task from Unbreak Now! to Normal.Jun 3 2019, 10:58 AM

Well, the flakiness of those particular tests seems to have been resolved by disabling them; I assume we’ll eventually want to fix and reenable them, but this task is certainly no longer UBN.

Tarrow added a subscriber: Tarrow.Thu, Aug 8, 10:12 AM

Adding another example of a flaky test:
https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-php70-docker/27043/console (https://phabricator.wikimedia.org/P8888) for posterity

Test failing is: Lexeme:Lemma can not save lemmas with redundant languages