Page MenuHomePhabricator

Refactor AbuseFilter WebdriverIO tests from sync to async mode
Closed, ResolvedPublic

Description

  • Follow instructions from AbuseFilter (Quickstart) to get AbuseFilter working locally.
  • Refactor WebdriverIO tests (in tests/selenium) from sync to async mode.
  • Push code to Gerrit.

Event Timeline

@Osamaahmed17: Please set appropriate project tags (in this case AbuseFilter). Thanks.

pwangai changed the task status from Open to In Progress.Mar 21 2023, 5:28 PM
pwangai claimed this task.

Change 901629 had a related patch set uploaded (by Pwangai; author: Pwangai):

[mediawiki/extensions/AbuseFilter@master] WIP selenium: Refactor WebdriverIO tests from sync to async mode

https://gerrit.wikimedia.org/r/901629

Change 901629 merged by jenkins-bot:

[mediawiki/extensions/AbuseFilter@master] selenium: Refactor WebdriverIO tests from sync to async mode

https://gerrit.wikimedia.org/r/901629

Hello @zeljkofilipin & @pwangai - For some reason this is making patches randomly fail (either on AbuseFilter or extensions that depend on AbuseFilter e.g. https://gerrit.wikimedia.org/r/c/mediawiki/extensions/OATHAuth/+/903681 - I know nothing about selenium, could you please take a look and see if there's an issue with the tests' code? Thank you.

Hey @MarcoAurelio , I am working on stablising the few flaky tests that keep randomly failing.

Change 904447 had a related patch set uploaded (by Pwangai; author: Pwangai):

[mediawiki/extensions/AbuseFilter@master] A recent change of AbuseFilter selenium tests from sync to async mode is making patches randomly fail. This is due to some flaky tests that sometimes fail and other times passes. In this patch I am changing the failing test to use the recommended expect-webdriverio format instead of assert() to see if this will improve the overall stability

https://gerrit.wikimedia.org/r/904447

Change 904447 abandoned by Pwangai:

[mediawiki/extensions/AbuseFilter@master] WIP selenium: fix unstable tests

Reason:

https://gerrit.wikimedia.org/r/904447

I'm not sure if this task should be closed as resolved. One of the tests that was previously working had to be disabled because it's now flaky. I would like to see that test re-enabled before calling this resolved. I will take a look at the test later.

@Daimona It would be helpful if you looked at the test to see if we could find a permanent solution. When I run the test multiple times locally on my machine, it doesn't fail, but somehow in CI it keeps failing. I have tried different approaches, but none seem to be stable in CI. Reopening the task.

Sorry it took me this long, for some reason selenium doesn't work on my local unless I set DISPLAY to :0 and hack the config to disable video recording. That said, the AbuseFilter tests are (very) sequential, I'm wondering if this could be causing the failure. Also, it looks like there is no recording of the failed test in CI, and it's passing for me locally, which makes debugging quite challenging.

Change 904796 had a related patch set uploaded (by Daimona Eaytoy; author: Daimona Eaytoy):

[mediawiki/extensions/AbuseFilter@master] tests: Improve selenium tests

https://gerrit.wikimedia.org/r/904796

Change 904796 merged by jenkins-bot:

[mediawiki/extensions/AbuseFilter@master] tests: Improve selenium tests

https://gerrit.wikimedia.org/r/904796

Change 904796 merged by jenkins-bot:

[mediawiki/extensions/AbuseFilter@master] tests: Improve selenium tests

https://gerrit.wikimedia.org/r/904796

Sorry @Daimona but this is making again unrelated changes for extensions that depend on AbuseFilter to fail: https://gerrit.wikimedia.org/r/c/mediawiki/extensions/WikimediaMessages/+/905659 :-(

Hmmm this is weird. The selenium suite was run 100 times for the patch above, and it passed 100/100. It could still be a transient issue, but I wonder if it might be caused by something (another extension, config etc.) that isn't there when running tests for AF. The video is also not very helpful, as it only shows the page.

Let's continue in T334001 and call this done.

Change 949566 had a related patch set uploaded (by Paladox; author: Pwangai):

[mediawiki/extensions/AbuseFilter@REL1_40] selenium: Refactor WebdriverIO tests from sync to async mode

https://gerrit.wikimedia.org/r/949566

Change 949571 had a related patch set uploaded (by Paladox; author: Daimona Eaytoy):

[mediawiki/extensions/AbuseFilter@REL1_40] tests: Improve selenium tests

https://gerrit.wikimedia.org/r/949571

Change 949566 merged by Krinkle:

[mediawiki/extensions/AbuseFilter@REL1_40] selenium: Refactor WebdriverIO tests from sync to async mode

https://gerrit.wikimedia.org/r/949566

Change 949571 merged by jenkins-bot:

[mediawiki/extensions/AbuseFilter@REL1_40] tests: Improve selenium tests

https://gerrit.wikimedia.org/r/949571

Change 955790 had a related patch set uploaded (by Jforrester; author: Pwangai):

[mediawiki/extensions/AbuseFilter@REL1_39] selenium: Refactor WebdriverIO tests from sync to async mode

https://gerrit.wikimedia.org/r/955790

Change 955790 merged by Jforrester:

[mediawiki/extensions/AbuseFilter@REL1_39] selenium: Refactor WebdriverIO tests from sync to async mode

https://gerrit.wikimedia.org/r/955790