Page MenuHomePhabricator
Create Task
Maniphest T196960

Quibble should have a way for extensions to opt out of core selenium browser tests
Closed, ResolvedPublic
Actions

Description

Some extensions, like Blackout, intentionally interfere with standard operation of MediaWiki and cause browser tests to fail. Extensions should have some way to signal to quibble that browser tests should be skipped for that repository.

Examples:

Note: all those extensions have been migrated to a Quibble job that passes --skip selenium.

AddThisInjects share\n in content
AccessControlDenies access by default
BlackoutIntentionally uses a black overlay over the content
CommentBoxInject content in article
ConfirmAccountwebdriver.io: miss #wpName2
EditSubpagesRequire login T196436
ForcePreview
OpenStackManagerattempts to auth with ldap which fails...
NewSignupPageOverhauls Special:Login
PremoderationT198952 archived , T157417 Trigger an error to notify edit has been put in a moderation Queue
PrivateDomains
SimpleSAMLphp
UnCaptcha"I am not a robot" check box on the create account page
Wikispeech<div id="ext-wikispeech-control-panel"> and alert() on edit
WikimediaIncubatorError: This page is unprefixed

See also T196436 which suggests to add a wdio test in extension to potentially tune some permissions. But that might be the wrong approach.

Details

Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
Update QA report for noselenium jobsHasharintegration/configmaster+2 -1
Migrate some extensions to Quibble without SeleniumHasharintegration/configmaster+13 -53
Quibble jobs that skip selenium entirelyHasharintegration/configmaster+58 -0
jjb: add support to pass arguments to quibbleHasharintegration/configmaster+2 -1
Skip selenium tests on some repositoriesHasharintegration/configmaster+16 -1
Customize query in gerrit

Related Objects
Search...

Event Timeline

Legoktm created this task.Jun 12 2018, 12:49 AM
zeljkofilipin added projects: User-zeljkofilipin, MediaWiki-Core-Tests.Jun 14 2018, 9:12 AM
zeljkofilipin moved this task from Inbox to Selenium on the MediaWiki-Core-Tests board.
hashar updated the task description. (Show Details)Jun 14 2018, 9:42 AM
hashar added a parent task: T183512: [EPIC] Migrate Mediawiki jobs from Nodepool to Docker.
hashar updated the task description. (Show Details)
hashar updated the task description. (Show Details)Jun 14 2018, 9:44 AM
hashar mentioned this in T196436: EditSubpages extension fails mediawiki/core webdriver.io tests.
hashar updated the task description. (Show Details)Jun 19 2018, 3:05 PM
hashar mentioned this in T183512: [EPIC] Migrate Mediawiki jobs from Nodepool to Docker.Jun 19 2018, 3:09 PM
hashar updated the task description. (Show Details)
hashar renamed this task from Quibble should have a way for extensions to opt out of core browser tests to Quibble should have a way for extensions to opt out of core selenium browser tests.Jun 22 2018, 5:18 PM
hashar updated the task description. (Show Details)
hashar updated the task description. (Show Details)Jun 25 2018, 8:15 PM
gerritbot subscribed.Jun 25 2018, 9:12 PM

Change 441984 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Skip selenium tests on some repositories

https://gerrit.wikimedia.org/r/441984

gerritbot added a project: Patch-For-Review.Jun 25 2018, 9:12 PM
hashar claimed this task.Jun 25 2018, 9:15 PM
hashar added a project: Continuous-Integration-Infrastructure (shipyard).
Restricted Application added a project: Release-Engineering-Team (Kanban). · View Herald TranscriptJun 25 2018, 9:15 PM
hashar moved this task from Backlog to In-progress on the Release-Engineering-Team (Kanban) board.Jun 25 2018, 9:15 PM
• Vvjjkkii renamed this task from Quibble should have a way for extensions to opt out of core selenium browser tests to 77aaaaaaaa.Jul 1 2018, 1:05 AM
• Vvjjkkii removed hashar as the assignee of this task.
• Vvjjkkii triaged this task as High priority.
• Vvjjkkii added projects: CheckUser, Connected-Open-Heritage-Batch-uploads (RAÄ-KMB_1_2017-02), Tamil-Sites, Gamepress, Hashtags, Jade, KartoEditor, Language-2018-Apr-June, New-Editor-Experiences, Mail, TCB-Team (now WMDE-TechWish).
• Vvjjkkii updated the task description. (Show Details)
• Vvjjkkii removed a subscriber: gerritbot.
• thcipriani raised the priority of this task from High to Needs Triage.Jul 1 2018, 7:41 PM
• thcipriani added a subscriber: gerritbot.
• thcipriani renamed this task from 77aaaaaaaa to Quibble should have a way for extensions to opt out of core selenium browser tests.Jul 1 2018, 7:41 PM
• thcipriani assigned this task to hashar.
• thcipriani updated the task description. (Show Details)
• thcipriani removed projects: TCB-Team (now WMDE-TechWish), Mail, New-Editor-Experiences, Language-2018-Apr-June, KartoEditor, Jade, Hashtags, Gamepress, Tamil-Sites, CheckUser.
hashar updated the task description. (Show Details)Jul 2 2018, 9:25 AM
hashar added a comment.Jul 2 2018, 3:42 PM

I thought about injecting --skip=selenium from the Zuul parameter file https://gerrit.wikimedia.org/r/#/c/integration/config/+/441984/ , but that means having a bunch of logic inside CI itself.

Timo / Kunal suggested to add a .quibble file in the extensions to be able to skip the tests easily but I am not a fan of that idea. That sounds like reimplementing Travis / Zuulv3 or what not.

After some reflection over the week-end, I think Quibble should be simply not run the selenium tests unless the extension has selenium tests. Multiple extensions fail solely because they are breaking expectations made by the very simple mediawiki/core tests. Blackout is a good example (it overlays a black page), or NewPageSignup which completely overhaul the Special:Login page.

So my idea is to check whether any file match **/tests/selenium/specs/**/*.js in the ZUUL_PROJECT that triggered the test, and if not, skip wdio / Selenium tests entirely.

gerritbot added a comment.Jul 2 2018, 3:44 PM

Change 441984 abandoned by Hashar:
Skip selenium tests on some repositories

Reason:
I am not a fan of my patch which add the logic in Zuul parameter_functions, that makes it hard for developer to figure out what is going on.

I am not a fan either of adding a quibble config file in extensions. I think it is going to be a nightmare eventually.

What we could do instead, is to just skip the Selenium tests when the repository does not have any. Lot of extensions are breaking the mediawiki/core tests that are intended for a stock install. Lets follow up on T196960#4387357 !

https://gerrit.wikimedia.org/r/441984

hashar updated the task description. (Show Details)Jul 2 2018, 3:45 PM
Lokal_Profil removed a project: Connected-Open-Heritage-Batch-uploads (RAÄ-KMB_1_2017-02).Jul 3 2018, 7:16 AM
zeljkofilipin moved this task from Backlog 🪒 to Quarter 👔 on the User-zeljkofilipin board.Jul 3 2018, 9:12 AM
hashar updated the task description. (Show Details)Jul 5 2018, 2:58 PM
hashar updated the task description. (Show Details)Jul 5 2018, 4:44 PM
hashar added a comment.Jul 5 2018, 4:49 PM

Talked quickly about it with Zeljko, maybe we can add some variants of the jobs that pass --skip selenium.

The mediawiki-quibble-{packages-source}-{database}-{php}-docker job template is intended for mediawiki/core and uses --skip composer-test to optimize the run time.

gerritbot added a comment.Jul 5 2018, 5:00 PM

Change 444024 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] jjb: add support to pass arguments to quibble

https://gerrit.wikimedia.org/r/444024

gerritbot added a comment.Jul 5 2018, 5:11 PM

Change 444028 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Quibble jobs that skip selenium entirely

https://gerrit.wikimedia.org/r/444028

gerritbot added a comment.Jul 6 2018, 10:08 AM

Change 444024 merged by jenkins-bot:
[integration/config@master] jjb: add support to pass arguments to quibble

https://gerrit.wikimedia.org/r/444024

gerritbot added a comment.Jul 6 2018, 10:10 AM

Change 444028 merged by jenkins-bot:
[integration/config@master] Quibble jobs that skip selenium entirely

https://gerrit.wikimedia.org/r/444028

gerritbot added a comment.Jul 6 2018, 11:00 AM

Change 444195 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Migrate some extensions to Quibble without Selenium

https://gerrit.wikimedia.org/r/444195

hashar updated the task description. (Show Details)Jul 6 2018, 11:15 AM
gerritbot added a comment.Jul 6 2018, 11:15 AM

Change 444195 merged by jenkins-bot:
[integration/config@master] Migrate some extensions to Quibble without Selenium

https://gerrit.wikimedia.org/r/444195

hashar triaged this task as Low priority.Jul 6 2018, 11:16 AM

In CI, I have migrated the extensions to use a Quibble job with --skip selenium. So that is less of a concern for the migration toward Docker/Quibble.

We probably still should find a better way to skip the tests.

hashar edited projects, added: Release-Engineering-Team (Backlog); removed: Release-Engineering-Team (Kanban), Continuous-Integration-Infrastructure (shipyard), Patch-For-Review.Jul 6 2018, 11:16 AM
gerritbot added a comment.Jul 6 2018, 11:46 AM

Change 444207 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Update QA report for noselenium jobs

https://gerrit.wikimedia.org/r/444207

gerritbot added a project: Patch-For-Review.Jul 6 2018, 11:46 AM
gerritbot added a comment.Jul 6 2018, 11:50 AM

Change 444207 merged by jenkins-bot:
[integration/config@master] Update QA report for noselenium jobs

https://gerrit.wikimedia.org/r/444207

hashar closed this task as Resolved.Jul 9 2018, 2:26 PM
hashar updated the task description. (Show Details)

Good enough for now. Solved by introducing a new set of Jenkins jobs that use --skip selenium.

hashar added a comment.Jul 17 2018, 4:31 PM

Note T198201 / https://gerrit.wikimedia.org/r/#/c/mediawiki/core/+/445971/ would let an extension inject settings.

Legoktm mentioned this in T225248: Consider moving browser based tests (Selenium and QUnit) to a non-voting pipeline.Jun 7 2019, 4:06 PM
hashar mentioned this in T250420: OpenStackManager should adjust selenium tests, rather than just not run them.May 3 2020, 9:10 PM
hashar mentioned this in T250418: WikimediaIncubator should adjust selenium tests, rather than just not run them.May 3 2020, 9:13 PM
Lokal_Profil mentioned this in T254942: Re-activate selenium tests.Jun 9 2020, 9:10 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits