Page MenuHomePhabricator

Find flaky Selenium tests
Open, Needs TriagePublic

Description

For start, create patches for each repository running the tests suite 10 times.

RepositoryGerritJenkins Job (failures)quibble-vendor-mysql-php74-selenium-docker (failures)wmf-quibble-selenium-php74-docker (failures)MediaWiki-Docker (T256239)
mediawiki/core721790mediawiki-quibble-selenium-vendor-mysql-php72-docker P30977 0/100P30976 2/100 ๐ŸšจP30863 0/100
mediawiki/extensions/AbuseFilter810352P30979 0/100P30980 0/100P30922 0/100
mediawiki/extensions/AdvancedSearch810354P30981 1/34 ๐Ÿšจ
mediawiki/extensions/CampaignEvents836815P35361 2/100 ๐Ÿšจ
mediawiki/extensions/CentralNotice
mediawiki/extensions/CirrusSearch
mediawiki/extensions/Cite810357P30982 0/100P30984 0/100P30924 0/100
mediawiki/extensions/CodeMirror810358P30985 0/100
mediawiki/extensions/Echo810359P30986 0/100P30987 0/100P30928 1/100 ๐Ÿšจ
mediawiki/extensions/ElectronPdfService810360P30988 0/100P30929 0/100
mediawiki/extensions/EntitySchema810364P30989 0/53P30942 0/100
mediawiki/extensions/FileImporter810366P30842 0/100P30843 0/100
mediawiki/extensions/GlobalWatchlist810370P30991 1/100 ๐ŸšจP30943 0/100
mediawiki/extensions/GrowthExperiments810371P30992 37/53 ๐ŸšจP30993 36/52 ๐Ÿšจ
mediawiki/extensions/Math810372P31827 0/100P31828 0/100P30944 0/100
mediawiki/extensions/MobileFrontend810373P31829 0/100P31830 0/100P30945 0/100
mediawiki/extensions/Newsletter810374P31836 0/100P30949 0/100
mediawiki/extensions/PageTriage993696P55812 6/100 ๐ŸšจP55813 8/89 ๐Ÿšจ
mediawiki/extensions/Popups810375P30964 0/100P30954 17/100 ๐Ÿšจ P30966 9/100 ๐Ÿšจ
mediawiki/extensions/ProofreadPage810377P31838 0/100P31839 0/100P30967 0/100
mediawiki/extensions/RelatedArticles810378P31840 0/100P30968 0/100
mediawiki/extensions/RevisionSlider810379P31842 0/32P30969 0/100
mediawiki/extensions/TemplateWizard810820P31843 2/100 ๐ŸšจP30972 8/100 ๐Ÿšจ
mediawiki/extensions/TwoColConflict810821P31844 0/17P31000 1/100 ๐Ÿšจ
mediawiki/extensions/VisualEditor807126P30857 0/100P30858 0/100P31410 0/100
mediawiki/extensions/Wikibase810831P31847 0/27P31849 0/24P31500 1/100 ๐Ÿšจ
mediawiki/extensions/WikibaseLexeme810823P31865 0/11
mediawiki/extensions/WikiLambda810824P31866 0/100P31428 0/100
mediawiki/extensions/Wikistories810827P31867 0/100P31889 0/100
mediawiki/skins/MinervaNeue810828quibble-vendor-mysql-php72-docker P31868 0/100P31869 0/100P31896 1/100 ๐Ÿšจ
wikibase/termbox810835
wikidata/query/gui810833generic-node14-browser-webdriver-docker P31715 0/100
wmde/wdio-wikibase

Notes

Details

SubjectRepoBranchLines +/-
mediawiki/extensions/CampaignEventsmaster+3 -1
mediawiki/coremaster+2 -2
mediawiki/extensions/PageTriagemaster+5 -5
mediawiki/extensions/PageTriagemaster+2 -1
mediawiki/extensions/VisualEditormaster+3 -2
mediawiki/coremaster+1 -1
mediawiki/skins/MinervaNeuemaster+3 -2
mediawiki/extensions/WikibaseLexememaster+3 -1
mediawiki/extensions/RelatedArticlesmaster+3 -2
mediawiki/extensions/MobileFrontendmaster+4 -2
mediawiki/extensions/CampaignEventsmaster+3 -2
mediawiki/extensions/FileImportermaster+3 -2
wikibase/termboxmaster+21 -0
mediawiki/extensions/Popupsmaster+3 -2
mediawiki/extensions/Citemaster+3 -2
mediawiki/extensions/AbuseFiltermaster+3 -2
mediawiki/extensions/CodeMirrormaster+3 -2
mediawiki/extensions/AdvancedSearchmaster+3 -1
mediawiki/extensions/ElectronPdfServicemaster+3 -2
mediawiki/extensions/GlobalWatchlistmaster+3 -2
mediawiki/extensions/EntitySchemamaster+3 -2
wikidata/query/guimaster+4 -1
wikimedia/portals/deploymaster+4 -1
mediawiki/extensions/Echomaster+3 -2
mediawiki/extensions/GrowthExperimentsmaster+2 -2
mediawiki/extensions/Wikistoriesmaster+3 -2
mediawiki/extensions/Mathmaster+3 -2
mediawiki/extensions/WikiLambdamaster+5 -2
mediawiki/extensions/Newslettermaster+3 -2
mediawiki/extensions/ProofreadPagemaster+3 -2
mediawiki/extensions/TemplateWizardmaster+3 -2
mediawiki/extensions/TwoColConflictmaster+3 -2
mediawiki/extensions/RevisionSlidermaster+3 -2
mediawiki/extensions/Wikibasemaster+4 -2
Show related patches Customize query in gerrit

Related Objects

StatusSubtypeAssignedTask
OpenNone
OpenNone
OpenNone
OpenNone
OpenNone
Resolvedkostajh
Resolvedovasileva
OpenNone
ResolvedOsamaahmed17
OpenNone
ResolvedWMDE-Fisch
DuplicateNone
Resolvedmatmarex
OpenNone
OpenNone
OpenBUG REPORTNone
OpenNone
ResolvedBUG REPORTNone
OpenNone
OpenNone
OpenNone

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Change 836815 had a related patch set uploaded (by Zfilipin; author: Zfilipin):

[mediawiki/extensions/CampaignEvents@master] WIP selenium: Find flaky tests

https://gerrit.wikimedia.org/r/836815

Change 836815 abandoned by Zfilipin:

[mediawiki/extensions/CampaignEvents@master] WIP selenium: Find flaky tests

Reason:

https://gerrit.wikimedia.org/r/836815

Change 810373 restored by Zfilipin:

[mediawiki/extensions/MobileFrontend@master] WIP selenium: Find flaky tests

https://gerrit.wikimedia.org/r/810373

Change 810373 abandoned by Zfilipin:

[mediawiki/extensions/MobileFrontend@master] WIP selenium: Find flaky tests

Reason:

https://gerrit.wikimedia.org/r/810373

Change 810378 restored by Zfilipin:

[mediawiki/extensions/RelatedArticles@master] WIP selenium: Find flaky tests

https://gerrit.wikimedia.org/r/810378

Change 810823 restored by Zfilipin:

[mediawiki/extensions/WikibaseLexeme@master] WIP selenium: Find flaky tests

https://gerrit.wikimedia.org/r/810823

Change 810378 abandoned by Zfilipin:

[mediawiki/extensions/RelatedArticles@master] WIP selenium: Find flaky tests

Reason:

https://gerrit.wikimedia.org/r/810378

Change 810828 restored by Zfilipin:

[mediawiki/skins/MinervaNeue@master] WIP selenium: Find flaky tests

https://gerrit.wikimedia.org/r/810828

Change 721790 restored by Zfilipin:

[mediawiki/core@master] WIP selenium: Find flaky tests

https://gerrit.wikimedia.org/r/721790

Change 810823 abandoned by Zfilipin:

[mediawiki/extensions/WikibaseLexeme@master] WIP selenium: Find flaky tests

Reason:

https://gerrit.wikimedia.org/r/810823

Change 810828 abandoned by Zfilipin:

[mediawiki/skins/MinervaNeue@master] WIP selenium: Find flaky tests

Reason:

https://gerrit.wikimedia.org/r/810828

Change 721790 abandoned by Zfilipin:

[mediawiki/core@master] WIP selenium: Find flaky tests

Reason:

https://gerrit.wikimedia.org/r/721790

Change 721790 restored by Zfilipin:

[mediawiki/core@master] WIP selenium: Find flaky tests

https://gerrit.wikimedia.org/r/721790

zeljkofilipin changed the task status from Open to In Progress.Dec 8 2022, 4:46 PM
zeljkofilipin claimed this task.

Change 866443 had a related patch set uploaded (by Zfilipin; author: Zfilipin):

[mediawiki/core@master] WIP selenium: Fix Special:Watchlist

https://gerrit.wikimedia.org/r/866443

zeljkofilipin changed the task status from In Progress to Open.Dec 9 2022, 11:58 AM
zeljkofilipin removed zeljkofilipin as the assignee of this task.

Change 807126 restored by Zfilipin:

[mediawiki/extensions/VisualEditor@master] WIP selenium: Run tests 100 times

https://gerrit.wikimedia.org/r/807126

https://integration.wikimedia.org/ci/job/wmf-quibble-selenium-php81-docker/1313/console#console-section-13 has some GrowthExperiment extension selenium tests time-out finding specific items (e.g.):

[0-0] Error in "add image.desktop: user can view image info and image details"
Error: element (".oo-ui-messageDialog-message") still not displayed after 5000ms

And also in general:

Error: Timeout of 60000ms exceeded. The execution in the test "add image mobile: user can close the image suggestion UI" took too long. Try to reduce the run time or increase your timeout for test specs (https://webdriver.io/docs/timeouts). (/workspace/src/extensions/GrowthExperiments/tests/selenium/specs/addimage.js)

After running the change through gate-and-submit again the same failures did not occur. Plus the selenium tests using other php versions passed without issues in the same attempt to get it through gate-and-submit.

Relatedly,

10:37:32 [Chrome 90.0.4430.212 linux #0-0] 1) add image mobile: user can close the image suggestion UI
10:37:32 [Chrome 90.0.4430.212 linux #0-0] Timeout of 60000ms exceeded. The execution in the test "add image mobile: user can close the image suggestion UI" took too long. Try to reduce the run time or increase your timeout for test specs (https://webdriver.io/docs/timeouts). (/workspace/src/extensions/GrowthExperiments/tests/selenium/specs/addimage.js)
10:37:32 [Chrome 90.0.4430.212 linux #0-0] Error: Timeout of 60000ms exceeded. The execution in the test "add image mobile: user can close the image suggestion UI" took too long. Try to reduce the run time or increase your timeout for test specs (https://webdriver.io/docs/timeouts). (/workspace/src/extensions/GrowthExperiments/tests/selenium/specs/addimage.js)
10:37:32 [Chrome 90.0.4430.212 linux #0-0]     at createTimeoutError (/workspace/src/extensions/GrowthExperiments/node_modules/mocha/lib/errors.js:498:15)
10:37:32 [Chrome 90.0.4430.212 linux #0-0]     at Test.Runnable._timeoutError (/workspace/src/extensions/GrowthExperiments/node_modules/mocha/lib/runnable.js:431:10)
10:37:32 [Chrome 90.0.4430.212 linux #0-0]     at Timeout.<anonymous> (/workspace/src/extensions/GrowthExperiments/node_modules/mocha/lib/runnable.js:246:24)
10:37:32 [Chrome 90.0.4430.212 linux #0-0]     at listOnTimeout (internal/timers.js:557:17)
10:37:32 [Chrome 90.0.4430.212 linux #0-0]     at processTimers (internal/timers.js:500:7)

Relatedly,

10:37:32 [Chrome 90.0.4430.212 linux #0-0] 1) add image mobile: user can close the image suggestion UI
10:37:32 [Chrome 90.0.4430.212 linux #0-0] Timeout of 60000ms exceeded. The execution in the test "add image mobile: user can close the image suggestion UI" took too long. Try to reduce the run time or increase your timeout for test specs (https://webdriver.io/docs/timeouts). (/workspace/src/extensions/GrowthExperiments/tests/selenium/specs/addimage.js)
10:37:32 [Chrome 90.0.4430.212 linux #0-0] Error: Timeout of 60000ms exceeded. The execution in the test "add image mobile: user can close the image suggestion UI" took too long. Try to reduce the run time or increase your timeout for test specs (https://webdriver.io/docs/timeouts). (/workspace/src/extensions/GrowthExperiments/tests/selenium/specs/addimage.js)
10:37:32 [Chrome 90.0.4430.212 linux #0-0]     at createTimeoutError (/workspace/src/extensions/GrowthExperiments/node_modules/mocha/lib/errors.js:498:15)
10:37:32 [Chrome 90.0.4430.212 linux #0-0]     at Test.Runnable._timeoutError (/workspace/src/extensions/GrowthExperiments/node_modules/mocha/lib/runnable.js:431:10)
10:37:32 [Chrome 90.0.4430.212 linux #0-0]     at Timeout.<anonymous> (/workspace/src/extensions/GrowthExperiments/node_modules/mocha/lib/runnable.js:246:24)
10:37:32 [Chrome 90.0.4430.212 linux #0-0]     at listOnTimeout (internal/timers.js:557:17)
10:37:32 [Chrome 90.0.4430.212 linux #0-0]     at processTimers (internal/timers.js:500:7)

Can you link to the build for this, please?

zeljkofilipin raised the priority of this task from Medium to Needs Triage.Feb 22 2023, 5:52 PM
zeljkofilipin moved this task from Q4 ๐Ÿ‘” to Backlog ๐Ÿช’ on the User-zeljkofilipin board.

Change 866443 abandoned by Zfilipin:

[mediawiki/core@master] WIP selenium: Fix Special:Watchlist

Reason:

https://gerrit.wikimedia.org/r/866443

Change 721790 abandoned by Zfilipin:

[mediawiki/core@master] WIP selenium: Find flaky tests

Reason:

https://gerrit.wikimedia.org/r/721790

Relatedly,

10:37:32 [Chrome 90.0.4430.212 linux #0-0] 1) add image mobile: user can close the image suggestion UI
10:37:32 [Chrome 90.0.4430.212 linux #0-0] Timeout of 60000ms exceeded. The execution in the test "add image mobile: user can close the image suggestion UI" took too long. Try to reduce the run time or increase your timeout for test specs (https://webdriver.io/docs/timeouts). (/workspace/src/extensions/GrowthExperiments/tests/selenium/specs/addimage.js)
10:37:32 [Chrome 90.0.4430.212 linux #0-0] Error: Timeout of 60000ms exceeded. The execution in the test "add image mobile: user can close the image suggestion UI" took too long. Try to reduce the run time or increase your timeout for test specs (https://webdriver.io/docs/timeouts). (/workspace/src/extensions/GrowthExperiments/tests/selenium/specs/addimage.js)
10:37:32 [Chrome 90.0.4430.212 linux #0-0]     at createTimeoutError (/workspace/src/extensions/GrowthExperiments/node_modules/mocha/lib/errors.js:498:15)
10:37:32 [Chrome 90.0.4430.212 linux #0-0]     at Test.Runnable._timeoutError (/workspace/src/extensions/GrowthExperiments/node_modules/mocha/lib/runnable.js:431:10)
10:37:32 [Chrome 90.0.4430.212 linux #0-0]     at Timeout.<anonymous> (/workspace/src/extensions/GrowthExperiments/node_modules/mocha/lib/runnable.js:246:24)
10:37:32 [Chrome 90.0.4430.212 linux #0-0]     at listOnTimeout (internal/timers.js:557:17)
10:37:32 [Chrome 90.0.4430.212 linux #0-0]     at processTimers (internal/timers.js:500:7)

Can you link to the build for this, please?

Created T334626 for this test

Change 807126 abandoned by Zfilipin:

[mediawiki/extensions/VisualEditor@master] WIP selenium: Run tests 100 times

Reason:

https://gerrit.wikimedia.org/r/807126

zeljkofilipin changed the task status from Open to In Progress.Jan 29 2024, 2:21 PM
zeljkofilipin claimed this task.
zeljkofilipin triaged this task as Low priority.
zeljkofilipin updated the task description. (Show Details)

Change 993696 had a related patch set uploaded (by Zfilipin; author: Zfilipin):

[mediawiki/extensions/PageTriage@master] WIP selenium: Find flaky tests

https://gerrit.wikimedia.org/r/993696

zeljkofilipin updated the task description. (Show Details)
zeljkofilipin updated the task description. (Show Details)

Change 994181 had a related patch set uploaded (by Zfilipin; author: Zfilipin):

[mediawiki/extensions/PageTriage@master] WIP selenium: Fix flaky test

https://gerrit.wikimedia.org/r/994181

Are you finding that missing awaits are a common cause of flaky Selenium tests? Would it make sense to set up some CI static code analysis that fails Selenium tests that don't await certain pre-programmed methods such as .click() and .isExisting()?

Change 993696 abandoned by Zfilipin:

[mediawiki/extensions/PageTriage@master] WIP selenium: Find flaky tests

Reason:

just a test

https://gerrit.wikimedia.org/r/993696

Change 994181 merged by jenkins-bot:

[mediawiki/extensions/PageTriage@master] selenium: Fix flaky test

https://gerrit.wikimedia.org/r/994181

zeljkofilipin changed the task status from In Progress to Open.Feb 9 2024, 1:24 PM
zeljkofilipin removed zeljkofilipin as the assignee of this task.
zeljkofilipin raised the priority of this task from Low to Needs Triage.

Change 721790 restored by Zfilipin:

[mediawiki/core@master] WIP selenium: Find flaky tests

https://gerrit.wikimedia.org/r/721790

zeljkofilipin changed the task status from Open to In Progress.Feb 19 2024, 12:47 PM
zeljkofilipin claimed this task.
zeljkofilipin triaged this task as Low priority.
zeljkofilipin changed the task status from In Progress to Open.Feb 19 2024, 5:29 PM
zeljkofilipin removed zeljkofilipin as the assignee of this task.
zeljkofilipin raised the priority of this task from Low to Needs Triage.

Change #721790 abandoned by Zfilipin:

[mediawiki/core@master] WIP selenium: Find flaky tests

Reason:

https://gerrit.wikimedia.org/r/721790

Change #1019084 had a related patch set uploaded (by Vaughn Walters; author: Vaughn Walters):

[mediawiki/extensions/CampaignEvents@master] selenium: Skipping test that fails on betacluster

https://gerrit.wikimedia.org/r/1019084

Change #1019084 merged by jenkins-bot:

[mediawiki/extensions/CampaignEvents@master] selenium: Skipping test that fails on betacluster

https://gerrit.wikimedia.org/r/1019084