Seen in https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-php72-docker/40820/console
09:12:25 5 passing (19.90s) 09:12:25 1 failing 09:12:25 09:12:25 1) Homepage can be disabled and re-enabled for new user: 09:12:25 The expression evaluated to a falsy value: 09:12:25 09:12:25 assert( !HomepagePage.homepage.isExisting() )
A "recheck" resulted in this test passing again.
| Subject | Repo | Branch | Lines +/- | |
|---|---|---|---|---|
| Disable flaky Selenium test | mediawiki/extensions/GrowthExperiments | master | +1 -1 |
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | None | T210726 Selenium framework cleanup | |||
| Open | None | T291254 Find flaky Selenium tests | |||
| Resolved | kostajh | T242953 Fix flaky test: Homepage can be disabled and re-enabled for new user |
Change 565234 had a related patch set uploaded (by Kosta Harlan; owner: Kosta Harlan):
[mediawiki/extensions/GrowthExperiments@master] Disable flaky Selenium test
Change 565234 merged by jenkins-bot:
[mediawiki/extensions/GrowthExperiments@master] Disable flaky Selenium test
I think this could use the same pattern as https://gerrit.wikimedia.org/r/c/mediawiki/extensions/GrowthExperiments/+/702893/7/tests/selenium/wdio.conf.js#52, where we use browser.waitUntil() to make sure the preference is saved before proceeding. @zeljkofilipin, what do you think?
I'm also OK to delete this test, as I don't think it gains us much in guarding against regressions or testing functionality.
@kostajh I'm fine with both. If you're not sure if the test is valuable, I would vote for it being deleted. Especially if it's flaky.