Page MenuHomePhabricator

Flaky test: Special:Watchlist "before each" hook for Special:Watchlist blocking tests in Vector repository
Open, HighPublicBUG REPORT

Description

e.g. https://integration.wikimedia.org/ci/job/wmf-quibble-selenium-php74-docker/10000/console

10:50:04 [Chrome 90.0.4430.212 linux #0-3] Running: Chrome (v90.0.4430.212) on linux
10:50:04 [Chrome 90.0.4430.212 linux #0-3] Session ID: a1b4a590-7fad-45a3-83ce-d4b9a66d080f
10:50:04 [Chrome 90.0.4430.212 linux #0-3]
10:50:04 [Chrome 90.0.4430.212 linux #0-3] » /tests/selenium/specs/watchlist.js
10:50:04 [Chrome 90.0.4430.212 linux #0-3] Special:Watchlist
10:50:04 [Chrome 90.0.4430.212 linux #0-3]    ? should show page with new edit
10:50:04 [Chrome 90.0.4430.212 linux #0-3]    ✖ "before each" hook for Special:Watchlist
10:50:04 [Chrome 90.0.4430.212 linux #0-3]
10:50:04 [Chrome 90.0.4430.212 linux #0-3] 1 failing (6.2s)
10:50:04 [Chrome 90.0.4430.212 linux #0-3]
10:50:04 [Chrome 90.0.4430.212 linux #0-3] 1) Special:Watchlist "before each" hook for Special:Watchlist
10:50:04 [Chrome 90.0.4430.212 linux #0-3] Can't call setValue on element with selector "#wpName1" because element wasn't found
10:50:04 [Chrome 90.0.4430.212 linux #0-3] Error: Can't call setValue on element with selector "#wpName1" because element wasn't found

Flaking often, blocking code reviews in Vector skin

Event Timeline

Jdlrobson triaged this task as Unbreak Now! priority.Dec 1 2022, 7:22 PM
kostajh renamed this task from Flakey test: Special:Watchlist "before each" hook for Special:Watchlist blocking tests in Vector repository to Flaky test: Special:Watchlist "before each" hook for Special:Watchlist blocking tests in Vector repository.Dec 7 2022, 12:19 PM
kostajh lowered the priority of this task from Unbreak Now! to High.

I don't think we usually use UBN for flaky tests? A workaround would be to skip the test (it.skip).

Change 865696 had a related patch set uploaded (by Zfilipin; author: Zfilipin):

[mediawiki/core@master] selenium: Skip flaky test

https://gerrit.wikimedia.org/r/865696

I don't think we usually use UBN for flaky tests?

I made it UBN as at the time it was flaking 100% of the time and blocking CI (similar to T324525) meaning it was impossible to merge code.
Side note: this doesn't seem to be blocking CI anymore. Just a mild annoyance so fine to lower priority.

Change 865696 merged by jenkins-bot:

[mediawiki/core@master] selenium: Skip flaky test

https://gerrit.wikimedia.org/r/865696

I have enabled Special:Watchlist should show page with new edit test in 866443.
I have ran all core tests 100 times in 721790.
There was only one failure. Page should be undoable failed once with Node is either not clickable or not an HTMLElement.
Special:Watchlist should show page with new edit never failed, in 100 runs.
I am not sure what to do. The test suite looks very robust, including Special:Watchlist should show page with new edit.
I can enable the test again next week, but I don't think we have a good way for me to monitor if it starts to fail again.