Page MenuHomePhabricator

Quibble should have a way for extensions to opt out of core selenium browser tests
Closed, ResolvedPublic

Description

Some extensions, like Blackout, intentionally interfere with standard operation of MediaWiki and cause browser tests to fail. Extensions should have some way to signal to quibble that browser tests should be skipped for that repository.

Examples:

Note: all those extensions have been migrated to a Quibble job that passes --skip selenium.

AddThisInjects share\n in content
AccessControlDenies access by default
BlackoutIntentionally uses a black overlay over the content
CommentBoxInject content in article
ConfirmAccountwebdriver.io: miss #wpName2
EditSubpagesRequire login T196436
ForcePreview
OpenStackManagerattempts to auth with ldap which fails...
NewSignupPageOverhauls Special:Login
PremoderationT198952 archived , T157417 Trigger an error to notify edit has been put in a moderation Queue
PrivateDomains
SimpleSAMLphp
UnCaptcha"I am not a robot" check box on the create account page
Wikispeech<div id="ext-wikispeech-control-panel"> and alert() on edit
WikimediaIncubatorError: This page is unprefixed

See also T196436 which suggests to add a wdio test in extension to potentially tune some permissions. But that might be the wrong approach.

Event Timeline

Legoktm created this task.Jun 12 2018, 12:49 AM
hashar updated the task description. (Show Details)Jun 19 2018, 3:05 PM
hashar renamed this task from Quibble should have a way for extensions to opt out of core browser tests to Quibble should have a way for extensions to opt out of core selenium browser tests.Jun 22 2018, 5:18 PM
hashar updated the task description. (Show Details)
hashar updated the task description. (Show Details)Jun 25 2018, 8:15 PM

Change 441984 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Skip selenium tests on some repositories

https://gerrit.wikimedia.org/r/441984

Vvjjkkii renamed this task from Quibble should have a way for extensions to opt out of core selenium browser tests to 77aaaaaaaa.Jul 1 2018, 1:05 AM
Vvjjkkii removed hashar as the assignee of this task.
Vvjjkkii triaged this task as High priority.
Vvjjkkii updated the task description. (Show Details)
Vvjjkkii removed a subscriber: gerritbot.
thcipriani raised the priority of this task from High to Needs Triage.Jul 1 2018, 7:41 PM
thcipriani added a subscriber: gerritbot.
thcipriani renamed this task from 77aaaaaaaa to Quibble should have a way for extensions to opt out of core selenium browser tests.Jul 1 2018, 7:41 PM
thcipriani assigned this task to hashar.
thcipriani updated the task description. (Show Details)
hashar updated the task description. (Show Details)Jul 2 2018, 9:25 AM
hashar added a comment.Jul 2 2018, 3:42 PM

I thought about injecting --skip=selenium from the Zuul parameter file https://gerrit.wikimedia.org/r/#/c/integration/config/+/441984/ , but that means having a bunch of logic inside CI itself.

Timo / Kunal suggested to add a .quibble file in the extensions to be able to skip the tests easily but I am not a fan of that idea. That sounds like reimplementing Travis / Zuulv3 or what not.

After some reflection over the week-end, I think Quibble should be simply not run the selenium tests unless the extension has selenium tests. Multiple extensions fail solely because they are breaking expectations made by the very simple mediawiki/core tests. Blackout is a good example (it overlays a black page), or NewPageSignup which completely overhaul the Special:Login page.

So my idea is to check whether any file match **/tests/selenium/specs/**/*.js in the ZUUL_PROJECT that triggered the test, and if not, skip wdio / Selenium tests entirely.

Change 441984 abandoned by Hashar:
Skip selenium tests on some repositories

Reason:
I am not a fan of my patch which add the logic in Zuul parameter_functions, that makes it hard for developer to figure out what is going on.

I am not a fan either of adding a quibble config file in extensions. I think it is going to be a nightmare eventually.

What we could do instead, is to just skip the Selenium tests when the repository does not have any. Lot of extensions are breaking the mediawiki/core tests that are intended for a stock install. Lets follow up on T196960#4387357 !

https://gerrit.wikimedia.org/r/441984

hashar updated the task description. (Show Details)Jul 2 2018, 3:45 PM
hashar updated the task description. (Show Details)Jul 5 2018, 2:58 PM
hashar updated the task description. (Show Details)Jul 5 2018, 4:44 PM
hashar added a comment.Jul 5 2018, 4:49 PM

Talked quickly about it with Zeljko, maybe we can add some variants of the jobs that pass --skip selenium.

The mediawiki-quibble-{packages-source}-{database}-{php}-docker job template is intended for mediawiki/core and uses --skip composer-test to optimize the run time.

Change 444024 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] jjb: add support to pass arguments to quibble

https://gerrit.wikimedia.org/r/444024

Change 444028 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Quibble jobs that skip selenium entirely

https://gerrit.wikimedia.org/r/444028

Change 444024 merged by jenkins-bot:
[integration/config@master] jjb: add support to pass arguments to quibble

https://gerrit.wikimedia.org/r/444024

Change 444028 merged by jenkins-bot:
[integration/config@master] Quibble jobs that skip selenium entirely

https://gerrit.wikimedia.org/r/444028

Change 444195 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Migrate some extensions to Quibble without Selenium

https://gerrit.wikimedia.org/r/444195

hashar updated the task description. (Show Details)Jul 6 2018, 11:15 AM

Change 444195 merged by jenkins-bot:
[integration/config@master] Migrate some extensions to Quibble without Selenium

https://gerrit.wikimedia.org/r/444195

hashar triaged this task as Low priority.Jul 6 2018, 11:16 AM

In CI, I have migrated the extensions to use a Quibble job with --skip selenium. So that is less of a concern for the migration toward Docker/Quibble.

We probably still should find a better way to skip the tests.

Change 444207 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Update QA report for noselenium jobs

https://gerrit.wikimedia.org/r/444207

Change 444207 merged by jenkins-bot:
[integration/config@master] Update QA report for noselenium jobs

https://gerrit.wikimedia.org/r/444207

hashar closed this task as Resolved.Jul 9 2018, 2:26 PM
hashar updated the task description. (Show Details)

Good enough for now. Solved by introducing a new set of Jenkins jobs that use --skip selenium.