AbuseFilter browser test errors
Open, Needs TriagePublic
Actions

Assigned To

None

Authored By

	Reedy
	Apr 4 2023, 5:58 PM

Description

https://gerrit.wikimedia.org/r/c/mediawiki/core/+/905680
https://integration.wikimedia.org/ci/job/wmf-quibble-selenium-php81-docker/5109/console

18:49:44 [0-0] AssertionError in "Filter editing.Trying to save a filter with bad data.data is retained if saving fails"
18:49:44 AssertionError [ERR_ASSERTION]: Expected values to be strictly equal:
18:49:44 + actual - expected
18:49:44 
18:49:44 + ''
18:49:44 - 'action === "edit"\n'
18:49:44     at Context.<anonymous> (/workspace/src/extensions/AbuseFilter/tests/selenium/specs/editingFilters.js:161:11)
18:49:44     at runMicrotasks (<anonymous>)
18:49:45 [0-0] RETRYING in chrome - /tests/selenium/specs/editingFilters.js
18:49:45 [0-0] RUNNING in chrome - /tests/selenium/specs/editingFilters.js
18:50:10 [0-0] AssertionError in "Filter editing.Trying to save a filter with bad data.data is retained if saving fails"
18:50:10 AssertionError [ERR_ASSERTION]: Expected values to be strictly equal:
18:50:10 + actual - expected
18:50:10 
18:50:10 + ''
18:50:10 - 'action === "edit"\n'
18:50:10     at Context.<anonymous> (/workspace/src/extensions/AbuseFilter/tests/selenium/specs/editingFilters.js:161:11)
18:50:10     at runMicrotasks (<anonymous>)
18:50:10 [0-0] FAILED in chrome - /tests/selenium/specs/editingFilters.js (1 retries)
18:50:10

Details

Subject	Repo	Branch	Lines +/-
Temporarily re-disable flaky selenium test	mediawiki/extensions/AbuseFilter	REL1_40	+1 -1
selenium: Switch to the base editor before reading the rules	mediawiki/extensions/AbuseFilter	master	+1 -0
Temporarily re-disable flaky selenium test	mediawiki/extensions/AbuseFilter	master	+1 -1

Customize query in gerrit

Related Objects

Mentioned In: T333966: message keys shown on beta and group0 wikis (test Wikidata, etc.)
T300790: Refactor AbuseFilter WebdriverIO tests from sync to async mode
Mentioned Here: T354729: AbuseFilter Selenium test fails in Wikibase(Lexeme) CI on REL1_40 (“data is retained if saving fails”)
T300790: Refactor AbuseFilter WebdriverIO tests from sync to async mode

Event Timeline

Reedy created this task.Apr 4 2023, 5:58 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 4 2023, 5:58 PM

TheresNoTime subscribed.Apr 4 2023, 6:05 PM

fwiw, issue was mentioned in T300790: Refactor AbuseFilter WebdriverIO tests from sync to async mode

Looks like it is indeed flaky and not a guaranteed failure, so doesn't need to be UBN/similar...

Daimona mentioned this in T300790: Refactor AbuseFilter WebdriverIO tests from sync to async mode.Apr 4 2023, 6:26 PM

Change 905699 had a related patch set uploaded (by Daimona Eaytoy; author: Daimona Eaytoy):

[mediawiki/extensions/AbuseFilter@master] selenium: Switch to the base editor before reading the rules

https://gerrit.wikimedia.org/r/905699

gerritbot added a project: Patch-For-Review.Apr 4 2023, 6:27 PM

Sooooooooooo, here's the situation:

If I run selenium with DISPLAY unset, it hangs when launching chrome (though I swear it worked a few days ago)
If I run selenium with DISPLAY=1, same as above
If I run selenium with DISPLAY=:0 (default), it works, but then it tries to record a video
I don't have ffmpeg installed, don't want to install it, and don't want videos. So I have to comment out the part that records videos
Even with DISPLAY=:0, chrome will crash with a SIGSEGV randomly while running the tests. This is not related to what test is running or anything. Most of the times it doesn't get to the end of the spec file.
The flaky test is the last test in its file, and I'm worried that skipping previous tests could falsify the result (yes, those tests are quite coupled, but oh well)
If the stars are aligned and nothing hangs or crashes, the test is passing locally

All this to say that I have no idea what makes it fail, and no way to verify that a fix works. Last time I ran the selenium tests 100 times in CI and they passed 100/100, yet today it fails. I suspect it's some timing issue that only appears when CI is overloaded or otherwise slow, but I have no way to confirm that. I hope that the patch above works, but I really don't know. As I've always said, selenium is black magic and I'm not a wizard.

pwangai subscribed.Apr 4 2023, 6:43 PM

This is blocking merges in Vector skin now.
Could this perhaps be disabled as a short term measure?

TheresNoTime mentioned this in T333966: message keys shown on beta and group0 wikis (test Wikidata, etc.).Apr 4 2023, 7:08 PM

Jdforrester-WMF subscribed.Apr 4 2023, 8:12 PM

In T334001#8756149, @Jdlrobson wrote:

This is blocking merges in Vector skin now.
Could this perhaps be disabled as a short term measure?

Yes, definitely, let me do that. It was already disabled, then I re-enabled it because it seemed to have been fixed... Also, FTR, the patch above doesn't fix the issue, since its first CI run failed immediately.

Change 905717 had a related patch set uploaded (by Daimona Eaytoy; author: Daimona Eaytoy):

[mediawiki/extensions/AbuseFilter@master] Temporarily re-disable flaky selenium test

https://gerrit.wikimedia.org/r/905717

Change 905717 merged by jenkins-bot:

[mediawiki/extensions/AbuseFilter@master] Temporarily re-disable flaky selenium test

https://gerrit.wikimedia.org/r/905717

Thanks!

ReleaseTaggerBot added a project: MW-1.41-notes (1.41.0-wmf.4; 2023-04-10).Apr 5 2023, 1:00 AM

Change 989484 had a related patch set uploaded (by Lucas Werkmeister (WMDE); author: Daimona Eaytoy):

[mediawiki/extensions/AbuseFilter@REL1_40] Temporarily re-disable flaky selenium test

https://gerrit.wikimedia.org/r/989484

Lucas_Werkmeister_WMDE merged a task: T354729: AbuseFilter Selenium test fails in Wikibase(Lexeme) CI on REL1_40 (“data is retained if saving fails”).Jan 10 2024, 4:36 PM

Lucas_Werkmeister_WMDE subscribed.

In T334001#8756102, @Daimona wrote:

All this to say that I have no idea what makes it fail, and no way to verify that a fix works. Last time I ran the selenium tests 100 times in CI and they passed 100/100, yet today it fails.

FWIW, in T354729 it seemed that the error was much more likely to occur in Wikibase(Lexeme)’s CI rather than in AbuseFilter’s own CI. (I was working with the REL1_40 branch, where I’ve now backported the test disablement per above; you might be able to get a similar result by uploading a Gerrit change to reenable the test on master, and testing empty Wikibase changes that pull this in via Depends-On.) But that’s still not a very pleasant debugging experience if the issue can’t be reproduced locally :S

Change 989484 merged by jenkins-bot: