Page MenuHomePhabricator
Create Task
Maniphest T324237

Flaky test: Special:Watchlist "before each" hook for Special:Watchlist blocking tests in Vector repository
Open, HighPublicBUG REPORT
Actions

Description

e.g. https://integration.wikimedia.org/ci/job/wmf-quibble-selenium-php74-docker/10000/console

10:50:04 [Chrome 90.0.4430.212 linux #0-3] Running: Chrome (v90.0.4430.212) on linux
10:50:04 [Chrome 90.0.4430.212 linux #0-3] Session ID: a1b4a590-7fad-45a3-83ce-d4b9a66d080f
10:50:04 [Chrome 90.0.4430.212 linux #0-3]
10:50:04 [Chrome 90.0.4430.212 linux #0-3] » /tests/selenium/specs/watchlist.js
10:50:04 [Chrome 90.0.4430.212 linux #0-3] Special:Watchlist
10:50:04 [Chrome 90.0.4430.212 linux #0-3]    ? should show page with new edit
10:50:04 [Chrome 90.0.4430.212 linux #0-3]    ✖ "before each" hook for Special:Watchlist
10:50:04 [Chrome 90.0.4430.212 linux #0-3]
10:50:04 [Chrome 90.0.4430.212 linux #0-3] 1 failing (6.2s)
10:50:04 [Chrome 90.0.4430.212 linux #0-3]
10:50:04 [Chrome 90.0.4430.212 linux #0-3] 1) Special:Watchlist "before each" hook for Special:Watchlist
10:50:04 [Chrome 90.0.4430.212 linux #0-3] Can't call setValue on element with selector "#wpName1" because element wasn't found
10:50:04 [Chrome 90.0.4430.212 linux #0-3] Error: Can't call setValue on element with selector "#wpName1" because element wasn't found

Flaking often, blocking code reviews in Vector skin

Details

ProjectBranchLines +/-Subject
mediawiki/coremaster+2 -1selenium: Skip flaky test
Customize query in gerrit

Related Objects
Search...

Event Timeline

Jdlrobson created this task.Dec 1 2022, 7:22 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptDec 1 2022, 7:22 PM
Jdlrobson triaged this task as Unbreak Now! priority.Dec 1 2022, 7:22 PM
Jdlrobson added subscribers: Jdforrester-WMF, zeljkofilipin.
Reedy added a project: MediaWiki-Watchlist.Dec 1 2022, 7:26 PM
Restricted Application added a project: Growth-Team. · View Herald TranscriptDec 1 2022, 7:26 PM
kostajh added a project: Quality-and-Test-Engineering-Team (QTE).Dec 7 2022, 12:18 PM
kostajh moved this task from Inbox to Triaged on the Growth-Team board.
kostajh added a subscriber: kostajh.

Looks similar to T324544: VisualEditor selenium tests sometimes fail.

kostajh renamed this task from Flakey test: Special:Watchlist "before each" hook for Special:Watchlist blocking tests in Vector repository to Flaky test: Special:Watchlist "before each" hook for Special:Watchlist blocking tests in Vector repository.Dec 7 2022, 12:19 PM
kostajh lowered the priority of this task from Unbreak Now! to High.

I don't think we usually use UBN for flaky tests? A workaround would be to skip the test (it.skip).

zeljkofilipin claimed this task.Dec 7 2022, 3:49 PM
zeljkofilipin added a project: User-zeljkofilipin.
zeljkofilipin moved this task from Inbox to Test engineering on the Quality-and-Test-Engineering-Team (QTE) board.
zeljkofilipin moved this task from Backlog 🪒 to In Progress 🎸 on the User-zeljkofilipin board.
gerritbot added a comment.Dec 7 2022, 4:49 PM

Change 865696 had a related patch set uploaded (by Zfilipin; author: Zfilipin):

[mediawiki/core@master] selenium: Skip flaky test

https://gerrit.wikimedia.org/r/865696

gerritbot added a project: Patch-For-Review.Dec 7 2022, 4:49 PM
Jdlrobson added a comment.Dec 7 2022, 4:54 PM
In T324237#8450318, @kostajh wrote:

I don't think we usually use UBN for flaky tests?

I made it UBN as at the time it was flaking 100% of the time and blocking CI (similar to T324525) meaning it was impossible to merge code.
Side note: this doesn't seem to be blocking CI anymore. Just a mild annoyance so fine to lower priority.

gerritbot added a comment.Dec 7 2022, 5:10 PM

Change 865696 merged by jenkins-bot:

[mediawiki/core@master] selenium: Skip flaky test

https://gerrit.wikimedia.org/r/865696

Maintenance_bot removed a project: Patch-For-Review.Dec 7 2022, 5:30 PM
ReleaseTaggerBot added a project: MW-1.40-notes (1.40.0-wmf.14; 2022-12-12).Dec 7 2022, 6:01 PM
zeljkofilipin added a parent task: T291254: Find flaky Selenium tests.Dec 9 2022, 11:35 AM
zeljkofilipin added a comment.Dec 9 2022, 11:56 AM

I have enabled Special:Watchlist should show page with new edit test in 866443.
I have ran all core tests 100 times in 721790.
There was only one failure. Page should be undoable failed once with Node is either not clickable or not an HTMLElement.
Special:Watchlist should show page with new edit never failed, in 100 runs.
I am not sure what to do. The test suite looks very robust, including Special:Watchlist should show page with new edit.
I can enable the test again next week, but I don't think we have a good way for me to monitor if it starts to fail again.

zeljkofilipin removed zeljkofilipin as the assignee of this task.Dec 15 2022, 11:47 AM
zeljkofilipin moved this task from In Progress 🎸 to Next 💉 on the User-zeljkofilipin board.
Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL