Page MenuHomePhabricator
Create Task
Maniphest T334634

Selenium test "Search.Clicking on a watchstar toggles the watchstar" is flaky
Open, Needs TriagePublic
Actions

Assigned To
None
Authored By
Umherirrender
Apr 12 2023, 11:27 PM
Tags
Referenced Files
F41541088: Search-Clicking-on-a-watchstar-toggles-the-watchstar-2023-11-27T19-22-07-003Z.mp4
Nov 27 2023, 7:44 PM
F41541087: Search-Clicking-on-a-watchstar-toggles-the-watchstar-2023-11-27T19-22-13-096Z.mp4
Nov 27 2023, 7:44 PM
F37094774: Search-Clicking-on-a-watchstar-toggles-the-watchstar-2023-06-05T21-21-14-481Z.mp4
Jun 5 2023, 10:24 PM
F37094770: Search-Clicking-on-a-watchstar-toggles-the-watchstar-2023-06-05T21-21-08-816Z.mp4
Jun 5 2023, 10:24 PM
Subscribers
Aklapper
Masumrezarock100
matmarex

Description

Failure on https://integration.wikimedia.org/ci/job/wmf-quibble-selenium-php81-docker/5355/console
The failure was for a gate-and-submit patch set of the cldr extension, the test definition is in MinervaNeue skin.

 Running: Chrome (v90.0.4430.212) on linux
 Session ID: 5c2d8780-17d0-4954-be5a-8c7cc8c89f3e

 » /tests/selenium/specs/search_loggedin.js
 Search
    ✖ Clicking on a watchstar toggles the watchstar

 1 failing (9.6s)

 1) Search Clicking on a watchstar toggles the watchstar
 Expected values to be strictly equal:
01:06:31 
01:06:31 false !== true
01:06:31       
01:06:31       actual expected
01:06:31       
01:06:31       falsetrue
01:06:31       
 AssertionError [ERR_ASSERTION]: Expected values to be strictly equal:

 false !== true

     at iAmLoggedIn (/workspace/src/skins/MinervaNeue/tests/selenium/features/step_definitions/common_steps.js:63:9)
     at async iAmLoggedIntoTheMobileWebsite (/workspace/src/skins/MinervaNeue/tests/selenium/features/step_definitions/common_steps.js:68:2)
     at async Context.<anonymous> (/workspace/src/skins/MinervaNeue/tests/selenium/specs/search_loggedin.js:22:3)

Video at https://integration.wikimedia.org/ci/job/wmf-quibble-selenium-php81-docker/5355/artifact/log/Search-Clicking-on-a-watchstar-toggles-the-watchstar-2023-04-12T23-06-15-187Z.mp4
Looks like wrong skin is loaded on the test

Details

SubjectRepoBranchLines +/-
Revert "Revert "Disable flakey test""mediawiki/skins/MinervaNeuemaster+2 -1
Revert "Disable flakey test"mediawiki/skins/MinervaNeuemaster+1 -2
Disable flakey testmediawiki/skins/MinervaNeuemaster+2 -1
Customize query in gerrit

Related Objects
Search...

Event Timeline

Umherirrender created this task.Apr 12 2023, 11:27 PM
Restricted Application added subscribers: Masumrezarock100, Aklapper. · View Herald TranscriptApr 12 2023, 11:27 PM
Umherirrender added a parent task: T291254: Find flaky Selenium tests.Apr 12 2023, 11:30 PM
Umherirrender added a comment.Apr 19 2023, 6:51 PM

Linking a recent issue as the files above may already deleted

https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-php74-selenium-docker/36307/console

matmarex subscribed.May 5 2023, 5:09 PM

I see this a lot. Recent examples:

Umherirrender added a comment.May 19 2023, 1:53 PM

Fresh one: https://integration.wikimedia.org/ci/job/wmf-quibble-selenium-php81-docker/6902/console

hashar subscribed.Jun 5 2023, 10:24 PM

Here are a couple videos captured for a Parsoid updated in mediawiki/vendor https://gerrit.wikimedia.org/r/c/mediawiki/vendor/+/927241

Search-Clicking-on-a-watchstar-toggles-the-watchstar-2023-06-05T21-21-08-816Z.mp4100 KBDownload

Search-Clicking-on-a-watchstar-toggles-the-watchstar-2023-06-05T21-21-14-481Z.mp4193 KBDownload

Extracted from https://integration.wikimedia.org/ci/job/wmf-quibble-selenium-php81-docker/7767/console

The MediaWiki error log is empty

21:21:08 [0-4] RUNNING in chrome - /tests/selenium/specs/search_loggedin.js
21:21:08 [0-5] RUNNING in chrome - /tests/selenium/specs/user_page.js
21:21:09 [0-5] PASSED in chrome - /tests/selenium/specs/user_page.js
21:21:09 [0-2] Nearby item will only appear in main menu if $wgMFNearby is configured
21:21:10 [0-2] PASSED in chrome - /tests/selenium/specs/mainmenu_loggedin.js
21:21:10 [0-6] RUNNING in chrome - /tests/selenium/specs/watchstar.js
21:21:12 [0-4] AssertionError in "Search.Clicking on a watchstar toggles the watchstar"
21:21:12 AssertionError [ERR_ASSERTION]: Expected values to be strictly equal:
21:21:12 
21:21:12 false !== true
21:21:12 
21:21:12     at iAmLoggedIn (/workspace/src/skins/MinervaNeue/tests/selenium/features/step_definitions/common_steps.js:63:9)
21:21:12     at async iAmLoggedIntoTheMobileWebsite (/workspace/src/skins/MinervaNeue/tests/selenium/features/step_definitions/common_steps.js:68:2)
21:21:12     at async Context.<anonymous> (/workspace/src/skins/MinervaNeue/tests/selenium/specs/search_loggedin.js:22:3)
21:21:13 [0-4] RETRYING in chrome - /tests/selenium/specs/search_loggedin.js
21:21:13 [0-4] RUNNING in chrome - /tests/selenium/specs/search_loggedin.js
21:21:17 [0-1] PASSED in chrome - /tests/selenium/specs/editor_wikitext_nosave.js
21:21:26 [0-4] AssertionError in "Search.Clicking on a watchstar toggles the watchstar"
21:21:26 AssertionError [ERR_ASSERTION]: Expected values to be strictly equal:
21:21:26 
21:21:26 false !== true
21:21:26 
21:21:26     at iAmLoggedIn (/workspace/src/skins/MinervaNeue/tests/selenium/features/step_definitions/common_steps.js:63:9)
21:21:26     at async iAmLoggedIntoTheMobileWebsite (/workspace/src/skins/MinervaNeue/tests/selenium/features/step_definitions/common_steps.js:68:2)
21:21:26     at async Context.<anonymous> (/workspace/src/skins/MinervaNeue/tests/selenium/specs/search_loggedin.js:22:3)
21:21:26 [0-4] FAILED in chrome - /tests/selenium/specs/search_loggedin.js (1 retries)
21:21:27 [0-6] PASSED in chrome - /tests/selenium/specs/watchstar.js
21:21:27

The test is retried and fails a second time (hence the two videos):

21:21:13 [0-4] RETRYING in chrome - /tests/selenium/specs/search_loggedin.js

There are some rate limitation reached, but they don't seem related.

...
2023-06-05 21:11:14 e7c5db630010 wikidb: MediaWiki\Permissions\RateLimiter::limit: limiting linkpurge rate for 127.0.0.1
2023-06-05 21:13:40 e7c5db630010 wikidb: MediaWiki\Permissions\RateLimiter::limit: limiting growthexperimentsuserimpacthandler rate for NewUser-0.42622400257586124-Iñtërnâtiônà lizætiøn
2023-06-05 21:13:43 e7c5db630010 wikidb: MediaWiki\Permissions\RateLimiter::limit: limiting stashbasehtml rate for NewUser-0.42622400257586124-Iñtërnâtiônà lizætiøn
2023-06-05 21:13:54 e7c5db630010 wikidb: MediaWiki\Permissions\RateLimiter::limit: limiting growthexperimentsuserimpacthandler rate for NewUser-0.16469190381122223-Iñtërnâtiônà lizætiøn
2023-06-05 21:13:57 e7c5db630010 wikidb: MediaWiki\Permissions\RateLimiter::limit: limiting stashbasehtml rate for NewUser-0.16469190381122223-Iñtërnâtiônà lizætiøn
2023-06-05 21:14:05 e7c5db630010 wikidb: MediaWiki\Permissions\RateLimiter::limit: limiting edit rate for NewUser-0.16469190381122223-Iñtërnâtiônà lizætiøn
2023-06-05 21:14:05 e7c5db630010 wikidb: MediaWiki\Permissions\RateLimiter::limit: limiting linkpurge rate for NewUser-0.16469190381122223-Iñtërnâtiônà lizætiøn
2023-06-05 21:17:15 e7c5db630010 wikidb: MediaWiki\Permissions\RateLimiter::limit: limiting edit rate for 127.0.0.1
...
matmarex added a comment.Jun 5 2023, 11:03 PM

So the user is supposed to be logged in and on the mobile website for this test, but in these videos, they are logged out and on desktop website.

There are currently several issues open about logging in on mobile not working, see e.g. T312042 and the other reports linked there. This is probably the root cause, and not the test itself.

Umherirrender unsubscribed.Jul 12 2023, 8:06 PM
gerritbot added a comment.Jul 27 2023, 4:46 PM

Change 942458 had a related patch set uploaded (by Jdlrobson; author: Jdlrobson):

[mediawiki/skins/MinervaNeue@master] Disable flakey test

https://gerrit.wikimedia.org/r/942458

gerritbot added a project: Patch-For-Review.Jul 27 2023, 4:46 PM
gerritbot added a comment.Jul 27 2023, 6:02 PM

Change 942458 merged by jenkins-bot:

[mediawiki/skins/MinervaNeue@master] Disable flakey test

https://gerrit.wikimedia.org/r/942458

Maintenance_bot removed a project: Patch-For-Review.Jul 27 2023, 6:10 PM
ReleaseTaggerBot added a project: MW-1.41-notes (1.41.0-wmf.20; 2023-08-01).Jul 27 2023, 7:00 PM
hashar unsubscribed.Jul 28 2023, 7:20 AM
matmarex added a comment.Nov 20 2023, 10:33 PM
In T334634#8904574, @matmarex wrote:

So the user is supposed to be logged in and on the mobile website for this test, but in these videos, they are logged out and on desktop website.

There are currently several issues open about logging in on mobile not working, see e.g. T312042 and the other reports linked there. This is probably the root cause, and not the test itself.

That is resolved now. Let's try to restore it, and see if it's still flaky.

gerritbot added a comment.Nov 20 2023, 10:34 PM

Change 975889 had a related patch set uploaded (by Bartosz Dziewoński; author: Bartosz Dziewoński):

[mediawiki/skins/MinervaNeue@master] Revert "Disable flakey test"

https://gerrit.wikimedia.org/r/975889

gerritbot added a project: Patch-For-Review.Nov 20 2023, 10:34 PM
gerritbot added a comment.Nov 20 2023, 11:27 PM

Change 975889 merged by jenkins-bot:

[mediawiki/skins/MinervaNeue@master] Revert "Disable flakey test"

https://gerrit.wikimedia.org/r/975889

Maintenance_bot removed a project: Patch-For-Review.Nov 20 2023, 11:30 PM
ReleaseTaggerBot added a project: MW-1.42-notes (1.42.0-wmf.7; 2023-11-28).Nov 21 2023, 1:00 AM
matmarex closed this task as Resolved.Nov 27 2023, 6:17 PM
matmarex claimed this task.
matmarex reopened this task as Open.Nov 27 2023, 7:44 PM

Ugh, it is still flaky, it just failed on https://gerrit.wikimedia.org/r/c/mediawiki/skins/MinervaNeue/+/975350 (https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-php80-docker/2248/console).

It looks like it's just using the desktop mode?

Search-Clicking-on-a-watchstar-toggles-the-watchstar-2023-11-27T19-22-13-096Z.mp4165 KBDownload

Search-Clicking-on-a-watchstar-toggles-the-watchstar-2023-11-27T19-22-07-003Z.mp4105 KBDownload

gerritbot added a comment.Nov 27 2023, 7:45 PM

Change 977621 had a related patch set uploaded (by Bartosz Dziewoński; author: Bartosz Dziewoński):

[mediawiki/skins/MinervaNeue@master] Revert "Revert "Disable flakey test""

https://gerrit.wikimedia.org/r/977621

gerritbot added a project: Patch-For-Review.Nov 27 2023, 7:45 PM
gerritbot added a comment.Nov 27 2023, 8:07 PM

Change 977621 merged by jenkins-bot:

[mediawiki/skins/MinervaNeue@master] Revert "Revert "Disable flakey test""

https://gerrit.wikimedia.org/r/977621

Maintenance_bot removed a project: Patch-For-Review.Nov 27 2023, 8:10 PM
matmarex removed matmarex as the assignee of this task.Nov 29 2023, 7:46 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL