Page MenuHomePhabricator
Create Task
Maniphest T199116

Quibble should run `npm install` and `npm run selenium-test` for each extension/skin that has Selenium tests
Closed, ResolvedPublic
Actions

Assigned To
awight
Authored By
zeljkofilipin
Jul 9 2018, 2:22 PM
Tags
Referenced Files
None
Subscribers
Addshore
Aklapper
awight
hashar
Jdforrester-WMF
Jdlrobson
Krinkle
View All 14 Subscribers
Tokens
"Party Time" token, awarded by Tarrow."Love" token, awarded by phuedx."Love" token, awarded by Lucas_Werkmeister_WMDE."Love" token, awarded by Addshore."Love" token, awarded by zeljkofilipin.

Description

Selenium framework should support multiple test runners (Mocha, Cucumber...) and assertion libraries (Assert, Chai...). So far all dependencies were managed from mediawiki core, but that has serious drawbacks (as @Krinkle points in T179190#4173431).

I have talked with @hashar and we have agreed that the way forward is to modify Quibble.

Workflow:

  • clone mediawiki core
  • clone extension(s)/skin(s)
  • find all wdio.conf.js files
  • cd into each extension/skin with wdio.conf.js file
  • run npm install
  • run npm run selenium-test

Running multiple npm install might slow Quibble down, so speeding up that part will be important.

Also, in order for the last step to work, each extension/skin has to have selenium-test script in package.json, but that will be resolved with T199113.

Note: see @Krinkle's comments at 429435.

See also: T193943: Selenium test job should install local dependencies before starting tests.

Details

SubjectRepoBranchLines +/-
selenium: stop running extensions/skins specsmediawiki/coremaster+1 -10
Stop running extension and skin browser tests with coremediawiki/coremaster+2 -12
selenium tests: Drop manual specification of extensions and skins' testsmediawiki/coremaster+1 -14
Revert "jjb: switch to quibble 0.0.34 containers"integration/configmaster+18 -18
Run node browser tests in each repointegration/quibblemaster+57 -17
Rename tests/selenium/ folder to "phpunit-selenium"mediawiki/extensions/PagedTiffHandlermaster+1 -1
selenium: Add jpeg-js to devDependenciesmediawiki/coremaster+1 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
phuedx added a comment.Feb 5 2019, 12:06 PM

Is there any chance that you did get time to look at this during All Hands, @hashar? I know it was very busy.

phuedx awarded a token.Feb 5 2019, 12:06 PM
hashar added a comment.Feb 5 2019, 1:08 PM

I haven't even opened my laptop over the week! That I guess tells we all were too busy discussing. Gotta now dig into implementing this in Quibble :]

Krinkle mentioned this in T199939: Audit tests/selenium/LocalSettings.php file aiming at possibly deprecating the feature.Feb 7 2019, 1:49 AM
Krinkle mentioned this in T198201: Quibble must include tests/selenium/LocalSettings.php (was ORES webdriver.io selenium test fail on CI due to lack of ORES server).Feb 7 2019, 1:55 AM
hashar removed hashar as the assignee of this task.Mar 19 2019, 8:40 PM

Sorry, I am definitely not working on this one :-(

zeljkofilipin mentioned this in T213268: Upgrade webdriverio to version 5 in mediawiki/core.Apr 1 2019, 10:49 AM
Jdlrobson mentioned this in T219920: CI tests disabled: Steps that require login are disabled and should be restored..Apr 3 2019, 10:22 PM
zeljkofilipin moved this task from Backlog 🪒 to Q4 👔 on the User-zeljkofilipin board.Apr 4 2019, 3:19 PM
zeljkofilipin mentioned this in T220035: Drop Ruby Selenium CI jobs; we don't support them any more.Apr 5 2019, 3:51 PM
Jdlrobson changed the task status from Stalled to Open.Apr 5 2019, 4:49 PM
Jdlrobson subscribed.

I'd like to unstall this one for visibility. This has been blocking my team from migrating our browser tests from Ruby to Node.js.
I the answer is we don't have the bandwidth to support Cucumber (which the timeframe of this ticket so far indicates) this would be useful to know and maybe my team can devise an alternate plan forward.

Resolving T219920 is a top priority for me, given this removes an important set of tests from our stack that are needed as we are changing code that touches them relating to our annual goal Advanced Mobile Contributions.

From my perspective T190710#5088918 is the main problem we need to unblock my team.

Michael subscribed.May 3 2019, 12:06 PM
gerritbot added a comment.May 10 2019, 2:25 PM

Change 509414 had a related patch set uploaded (by Zfilipin; owner: Zfilipin):
[mediawiki/core@master] Selenium: add jpeg-js to devDependencies

https://gerrit.wikimedia.org/r/509414

gerritbot added a project: Patch-For-Review.May 10 2019, 2:25 PM
gerritbot added a comment.May 10 2019, 5:19 PM

Change 509414 merged by jenkins-bot:
[mediawiki/core@master] selenium: Add jpeg-js to devDependencies

https://gerrit.wikimedia.org/r/509414

ReleaseTaggerBot added a project: MW-1.34-notes (1.34.0-wmf.5; 2019-05-14).May 10 2019, 6:00 PM
Krinkle mentioned this in T223150: Echo Selenium tests - Build failure - Cannot find module wdio-mediawiki/Util.May 13 2019, 6:01 PM
awight mentioned this in T223431: Need a mechanism for injecting a LocalSettings.php fragment before nodejs browser tests.May 16 2019, 10:27 AM
awight subscribed.May 16 2019, 10:47 AM

I'm going to take a brief look at this. The changes will need to be made in several steps, AIUI:

  • Patch quibble to run npm install && npm run selenium-test in the extension-or-skin root directory.
  • Build a new quibble image
  • Patch integration-config to use the new quibble image.
  • Tests will be duplicated at this point. We should verify this--each repo with tests/selenium/specs/*.js should also have a selenium-test script listed in package.json.
  • Remove extensions and skins from search patch in mw-core wdio.conf.js
awight added a comment.May 16 2019, 10:52 AM

Reading the task description again, I see that my understanding was slightly different than the original suggestion. The request is to run *all* extension and skin tests.

gerritbot added a comment.May 16 2019, 11:29 AM

Change 510709 had a related patch set uploaded (by Awight; owner: Awight):
[integration/quibble@master] [WIP] Run node browser tests in each repo

https://gerrit.wikimedia.org/r/510709

awight mentioned this in rQUIBBLE275fc65c1b57: [WIP] Run node browser tests in each repo.May 16 2019, 11:31 AM
awight mentioned this in rQUIBBLE89e8473cad97: Run node browser tests in each repo.May 16 2019, 12:01 PM
awight mentioned this in rQUIBBLE9dea5826ab75: Run node browser tests in each repo.May 16 2019, 12:15 PM
awight moved this task from Enhancement to In progress on the Quibble board.May 16 2019, 12:58 PM

@hashar This is ready for review, and I've smoke-tested locally. The big question IMO is whether I've implemented the desired functionality, this will run npm install && npm run selenium-test in each of self.dependencies which includes a tests/selenium.

awight mentioned this in T223967: Tangential CI work to support Tech Wishes.May 21 2019, 9:28 AM
gerritbot added a comment.May 22 2019, 2:22 PM

Change 511893 had a related patch set uploaded (by Awight; owner: Awight):
[mediawiki/core@master] Stop running extension and skin browser tests with core

https://gerrit.wikimedia.org/r/511893

gerritbot added a comment.May 22 2019, 2:39 PM

Change 511898 had a related patch set uploaded (by Thiemo Kreuz (WMDE); owner: Thiemo Kreuz (WMDE)):
[mediawiki/extensions/PagedTiffHandler@master] Rename tests/selenium/ folder to "phpunit-selenium"

https://gerrit.wikimedia.org/r/511898

gerritbot added a comment.May 23 2019, 10:09 AM

Change 511898 merged by jenkins-bot:
[mediawiki/extensions/PagedTiffHandler@master] Rename tests/selenium/ folder to "phpunit-selenium"

https://gerrit.wikimedia.org/r/511898

awight mentioned this in T222200: Run browser tests as part of "npm test" for wikidata/query/gui in CI.May 23 2019, 1:06 PM
ReleaseTaggerBot edited projects, added MW-1.34-notes (1.34.0-wmf.7; 2019-05-28); removed MW-1.34-notes (1.34.0-wmf.5; 2019-05-14).May 25 2019, 2:01 AM
awight added a subscriber: Jdforrester-WMF.May 27 2019, 10:20 AM

@Jdforrester-WMF made the suggestion that we block on T211784, so that selenium tests are disabled for the Node 10 upgrade.

We're interested in helping with this migration if possible, but first I wanted to discuss with the RelEng team.

If others agree that this patch as written is too dangerous and we should wait for browser tests to be disabled, I could make the quibble patch conditional on a flag or on configuration, so that it's possible to run the new logic only for experimental or specific jobs. Another alternative would be to disable browser tests just for this change--independent of the Node 10 work since that's not an explicit dependency for anything we're doing here.

awight claimed this task.May 29 2019, 10:18 AM
awight added a comment.Jun 4 2019, 9:47 AM

According to @Jdforrester-WMF's comment, we're waiting to merge these patches before this task can proceed.

Given the current wdio is incompatible with node10, upgrading to node10 *after* doing per-repo versions of wdio means we'll need to urgently unbreak bump dozens of repos with must-merge patches, instead of just MW core. I'm not sure that's a deal-breaker, but we really need to Just Do It™ at this point.

awight added a parent task: T190829: Test WikiTextEditor class with browser tests..Jun 4 2019, 9:47 AM
Jdforrester-WMF added a comment.Jun 4 2019, 4:20 PM

Should now be good to go.

awight mentioned this in T225330: Commit package-lock.json files everywhere.Jun 7 2019, 9:02 PM
awight added a comment.Jun 7 2019, 9:09 PM

There are at least two easy speedups for npm install, the first is documented in subtask T225330: Commit package-lock.json files everywhere, which just became possible thanks to the npm 10 upgrade. The second is --prefer-offline, which only fetches packages when they are missing from the local cache. We should experiment with both.

awight mentioned this in rQUIBBLE311454c7b82c: Run node browser tests in each repo.Jun 7 2019, 10:22 PM
awight mentioned this in rQUIBBLE805ff996f803: Run node browser tests in each repo.Jun 7 2019, 10:25 PM
awight mentioned this in rQUIBBLEe5c5db32a7bf: Run node browser tests in each repo.Jun 8 2019, 10:08 PM
hashar reopened subtask T225330: Commit package-lock.json files everywhere as Open.Jun 12 2019, 12:49 PM
awight mentioned this in T226869: Run browser tests in parallel.Jun 28 2019, 8:58 PM
greg edited projects, added Release-Engineering-Team-TODO (201907); removed Release-Engineering-Team (Kanban).Jul 1 2019, 9:25 PM
greg moved this task from INBOX to Ready on the Release-Engineering-Team-TODO (201907) board.Jul 1 2019, 9:27 PM
awight added a comment.Jul 3 2019, 9:48 AM

I've adjusted the patch to detect selenium tests based on the scripts in package.json, rather than testing for directory tests/selenium existence, so we'll be compatible with Wikibase and other unusual hierarchies.

awight mentioned this in T227321: 20% things for 26.June sprint.Jul 5 2019, 11:54 AM
greg edited projects, added Release-Engineering-Team-TODO; removed Release-Engineering-Team-TODO (201907).Jul 6 2019, 4:33 AM
greg moved this task from Should be empty (use Release-Engineering-Team) to Watching/External on the Release-Engineering-Team-TODO board.Jul 6 2019, 5:34 AM
Tarrow rescinded a token.Jul 9 2019, 1:48 PM
Tarrow awarded a token.
Tarrow mentioned this in T225196: Run Termbox browser tests as part of Wikibase per-patch browser tests jobs.Jul 9 2019, 1:55 PM
zeljkofilipin moved this task from Q4 👔 to Watching 📺 on the User-zeljkofilipin board.Jul 11 2019, 2:27 PM
awight added a comment.EditedJul 15 2019, 12:44 PM

Friends, I see that this task is in a few "external / watching" columns, so I want to clarify the status as I understand it. I've written a patch which should provide the needed functionality, and I believe it's both a safe migration and might not damage performance by much, since I've disabled npm's default cache-refresh network calls.

What we need now is code review of the patch.

Once this is merged, I have some more questions about a new mechanism for injecting snippets into LocalSettings.php, but that's follow-up work.

awight mentioned this in T227748: 20% things for 9.June sprint.Jul 16 2019, 8:21 AM
• Pablo-WMDE subscribed.Jul 16 2019, 9:53 AM
• Pablo-WMDE mentioned this in T228523: Integrate browser tests with CI.Jul 19 2019, 3:08 PM
WMDE-leszek subscribed.Jul 22 2019, 3:38 PM
Jdlrobson added a comment.Jul 22 2019, 7:36 PM

@zeljkofilipin would you be the right person to review for @awight ?

hashar added a comment.Jul 24 2019, 3:32 PM

It took me a few tries to reach out to @awight patch but I have finally managed to do a proper review of https://gerrit.wikimedia.org/r/#/c/integration/quibble/+/510709/ today :] It is almost ready to be merged!

greg added a project: Release-Engineering-Team (CI & Testing services).Jul 24 2019, 3:50 PM
gerritbot added a comment.Jul 25 2019, 2:33 PM

Change 510709 merged by jenkins-bot:
[integration/quibble@master] Run node browser tests in each repo

https://gerrit.wikimedia.org/r/510709

hashar mentioned this in rQUIBBLEb2f9e3630466: Run node browser tests in each repo.Jul 25 2019, 2:36 PM
Stashbot mentioned this in T193824: Determine a standard way of installing MediaWiki lib/extension dependencies within containers.Jul 25 2019, 2:52 PM
Stashbot mentioned this in T87781: Split mediawiki tests into unit and integration tests.

Mentioned in SAL (#wikimedia-releng) [2019-07-25T14:52:18Z] <hashar> Tagged Quibble 0.0.33 b2f9e36 | T193824 T87781 T199116

hashar added a subtask: T229033: wdio-cucumber-framework fails on NodeJS due to fibers@2.x under nodejs10.Jul 25 2019, 6:12 PM

The feature has been implemented by @awight and I have released it today with Quibble 0.0.33 (which is broken so really use 0.0.34).

I have deployed it on CI and that worked for MinervaNeue but fails on CirrusSearch/MobileFrontend due to wdio-cucumber-framework. T229033 is to update it.

Jdforrester-WMF closed subtask T229033: wdio-cucumber-framework fails on NodeJS due to fibers@2.x under nodejs10 as Resolved.Jul 25 2019, 9:15 PM
hashar reopened subtask T229033: wdio-cucumber-framework fails on NodeJS due to fibers@2.x under nodejs10 as Open.Jul 25 2019, 10:03 PM
gerritbot added a comment.Jul 25 2019, 10:09 PM

Change 525669 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Revert "jjb: switch to quibble 0.0.34 containers"

https://gerrit.wikimedia.org/r/525669

gerritbot added a comment.Jul 25 2019, 10:12 PM

Change 525669 merged by jenkins-bot:
[integration/config@master] Revert "jjb: switch to quibble 0.0.34 containers"

https://gerrit.wikimedia.org/r/525669

hashar closed subtask T229065: Quibble extension selenium test fails with: gyp ERR! stack Error: Can't find Python executable "python", you can set the PYTHON env variable. as Resolved.Jul 25 2019, 11:05 PM
hashar added a comment.Jul 25 2019, 11:08 PM

Implemented by @awight and released with 0.0.34. I have missed a few things after review and had to add a few hotfixes but overall it seems to be running fine now.

I guess now in tests/selenium/wdio.conf.js we can remove the various wildcard:

relPath( './extensions/*/tests/selenium/specs/**/*.js' ),
relPath( './extensions/VisualEditor/modules/ve-mw/tests/selenium/specs/**/*.js' ),
relPath( './extensions/Wikibase/repo/tests/selenium/specs/**/*.js' ),
relPath( './skins/*/tests/selenium/specs/**/*.js' )

Replaced by having to do: cd extensions/Foobar && npm run-script selenium-test.

Right now Quibble runs webdriver.io for mediawiki/core which glob all specs from extensions and skins AND it also run the tests via selenium-test. Thus the extensions / skins are run twice.

hashar closed this task as Resolved.Jul 25 2019, 11:20 PM
gerritbot added a comment.Jul 25 2019, 11:38 PM

Change 525692 had a related patch set uploaded (by Hashar; owner: Hashar):
[mediawiki/core@master] selenium: stop running extensions/skins specs

https://gerrit.wikimedia.org/r/525692

gerritbot added a comment.Jul 25 2019, 11:39 PM

Change 525693 had a related patch set uploaded (by Jforrester; owner: Jforrester):
[mediawiki/core@master] selenium tests: Drop manual specification of extensions and skins' tests

https://gerrit.wikimedia.org/r/525693

gerritbot added a comment.Jul 25 2019, 11:45 PM

Change 525693 abandoned by Jforrester:
selenium tests: Drop manual specification of extensions and skins' tests

Reason:
Antoine beat me by a few seconds. ;-)

https://gerrit.wikimedia.org/r/525693

gerritbot added a comment.Jul 31 2019, 8:25 AM

Change 511893 abandoned by Awight:
Stop running extension and skin browser tests with core

https://gerrit.wikimedia.org/r/511893

gerritbot added a comment.Aug 5 2019, 2:23 PM

Change 525692 merged by jenkins-bot:
[mediawiki/core@master] selenium: stop running extensions/skins specs

https://gerrit.wikimedia.org/r/525692

ReleaseTaggerBot edited projects, added MW-1.34-notes (1.34.0-wmf.17; 2019-08-06); removed MW-1.34-notes (1.34.0-wmf.7; 2019-05-28).Aug 5 2019, 3:00 PM
Maintenance_bot removed a project: Patch-For-Review.Aug 5 2019, 3:10 PM
awight added a comment.Aug 5 2019, 8:58 PM

I'm auditing a recent gate-and-submit job to sanity check the wdio.conf.js changes to mediawiki-core, and found this concerning line,

00:12:17.444 WARNING:backend.ChromeWebDriver:[1565037144.160][SEVERE]: bind() failed: Cannot assign requested address (99)

That looks like we're failing to manage the webdriver correctly.

Meanwhile, everything else looks good. Spot-checking a few test cases, each test is now run exactly once, so the core change is validated.

The separate browser tests are parsed into sections by Jenkins, which is a nice bonus.

ReleaseTaggerBot edited projects, added MW-1.34-notes (1.34.0-wmf.19; 2019-08-20); removed MW-1.34-notes (1.34.0-wmf.17; 2019-08-06).Aug 6 2019, 6:01 PM
hashar changed the status of subtask T229033: wdio-cucumber-framework fails on NodeJS due to fibers@2.x under nodejs10 from Open to Stalled.Aug 19 2019, 8:29 AM
hashar added a comment.Aug 22 2019, 7:11 AM
In T199116#5394564, @awight wrote:

I'm auditing a recent gate-and-submit job to sanity check the wdio.conf.js changes to mediawiki-core, and found this concerning line,

00:12:17.444 WARNING:backend.ChromeWebDriver:[1565037144.160][SEVERE]: bind() failed: Cannot assign requested address (99)

That looks like we're failing to manage the webdriver correctly.

I guess because somehow we try to start chomewebdriver a second time? That is worth a task :-]

Meanwhile, everything else looks good. Spot-checking a few test cases, each test is now run exactly once, so the core change is validated.

The separate browser tests are parsed into sections by Jenkins, which is a nice bonus.

Great! Thank you to have double checked :-]

hashar mentioned this in T232626: CI mediawiki/core run times have increased since July 26th, 2019.Sep 11 2019, 9:07 PM
awight removed a parent task: T190829: Test WikiTextEditor class with browser tests..Oct 22 2019, 12:01 PM
hashar closed subtask T229033: wdio-cucumber-framework fails on NodeJS due to fibers@2.x under nodejs10 as Resolved.Jun 14 2023, 1:02 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL