Page MenuHomePhabricator

Minerva Ruby and Node.js browser tests running side by side
Closed, ResolvedPublic5 Story Points

Description

Status

Blocked on T199116: Quibble should run `npm install` and `npm run selenium-test` for each extension/skin that has Selenium tests.

Description

To make migrating browser tests from Ruby to Node.js a more straight-forward affair we will setup the infrastructure for running both Ruby and Node.js browser tests. We will port over our most simple browser test to demonstrate it is working.

Once this is in place, we will be able to easily, safely and gradually port over our browser tests.

Acceptance criteria

  • Minerva has two folders - 'browser' (containing the existing Ruby tests) and 'selenium' (containing the new Node.js tests)
  • Make sure https://gerrit.wikimedia.org/r/#/c/421961/ has been merged to simplify implementation
  • The categories.feature browser is moved from 'browser' to 'selenium'. We will retain the existing cucumber notation and .feature file and provide step_definitions written in Node.js
  • Node.js browser tests and Ruby browser tests run with every commit
  • A selenium-MinervaNeue-jessie test is listed on https://integration.wikimedia.org/ci/view/Reading-Web/ which runs the Node.js browser tests as well as the selenium-MinervaNeue test that runs the Ruby browser tests.

TODO

๐Ÿ T199116: Quibble should run `npm install` and `npm run selenium-test` for each extension/skin that has Selenium tests
๐Ÿ 423648 Adding initial webdriver.io test
๐Ÿ 425422 Adding "diff.feature" selenium test

Run tests

~/Documents/gerrit/mediawiki/vagrant$ vagrant up
...

~/Documents/gerrit/mediawiki/vagrant$ vagrant roles enable minerva
Ok. Run `vagrant provision` to apply your changes.

~/Documents/gerrit/mediawiki/vagrant$ vagrant provision
...

~/Documents/gerrit/mediawiki/vagrant$ cd mediawiki/skins/MinervaNeue/

~/Documents/gerrit/mediawiki/vagrant/mediawiki/skins/MinervaNeue$ git review -d 423648
...

~/Documents/gerrit/mediawiki/vagrant/mediawiki/skins/MinervaNeue$ npm i
...

~/Documents/gerrit/mediawiki/vagrant/mediawiki/skins/MinervaNeue$ npm run selenium
...
Number of specs: 2
13 passing (51.20s)
sh: line 1:  7831 Terminated: 15          chromedriver --url-base=/wd/hub --port=4444

Related Objects

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

I don't know if the Selenium CI jobs are set up use testing frameworks other than mocha. In the patch had to extend the mediawiki-core wdio.conf.js file to specify 'cucumber' as the testing framework, but I don't think CI picks up that file.

At the moment, CI only runs Mocha tests.

CI is wired up to scan a specs folder for JS files but /home/jenkins/workspace/mediawiki-core-qunit-selenium-jessie/src/skins/*/tests/selenium/specs/**/*.js did not match any file

I'll take a look at how to make the job run both Mocha and Cucumber. Webdriverio support both, but I do not think I have ever tried to run both at the same time. There should be no technical difficulties. In Ruby world it was not unusual that a project has both RSpec and Cucumber tests, so I guess it is similar in Node.js with Mocha and Cucumber.

We're using features here so CI will needed to be able to deal with this different setup (which means loading Minerva's config). Since CirrusSearch is also using this, this would be helpful so they can stop using Cindy and rely on Jenkins CI.

CirrusSearch is a different story. It's not Cucumber that makes it difficult to run Selenium test, it's because CI at the moment does not support CirrusSearch. :)

Zeljko says this should be doable. He's going to take a look at it tomorrow and report back. He will do it in core so other extensions can make use of this - doing this in MobileFrontend should be pretty straightforward. Yay!

Update: webdriverio does not support running both mocha and cucumber at the same time. I am working on a solution to run them separately. See T179190 for details. I estimate to have something working tomorrow (Saturday).

zeljkofilipin added a comment.EditedApr 10 2018, 4:54 PM

Apologies for the delay, this proved to be a bit more complicated than I thought. ๐Ÿค” I have created a few patches while working on this, see topic:T179190. Reviews are welcome.

TLDR I can run Minerva Cucumber tests from Minerva folder, but not from core. ๐Ÿ˜•

Provisioning:

~/Documents/gerrit/mediawiki/vagrant$ vagrant up
...

~/Documents/gerrit/mediawiki/vagrant$ vagrant roles enable minerva
Ok. Run `vagrant provision` to apply your changes.

~/Documents/gerrit/mediawiki/vagrant$ vagrant provision
...

Running tests from Minerva:

~/Documents/gerrit/mediawiki/vagrant$ cd mediawiki/skins/MinervaNeue/

~/Documents/gerrit/mediawiki/vagrant/mediawiki/skins/MinervaNeue$ git review -d 423648
...

~/Documents/gerrit/mediawiki/vagrant/mediawiki/skins/MinervaNeue$ npm i
...

~/Documents/gerrit/mediawiki/vagrant/mediawiki/skins/MinervaNeue$ npm run selenium
...
Number of specs: 2
13 passing (51.20s)
sh: line 1:  7831 Terminated: 15          chromedriver --url-base=/wd/hub --port=4444

So far so good.

No. That is the overall epic. This task is complete when we have one test ported over.

I am doing something wrong in the core patch (424609). I am not sure where ERROR: Cannot read property 'split' of undefined comes from:

~/Documents/gerrit/mediawiki/vagrant/mediawiki/skins/MinervaNeue$ cd ../..

~/Documents/gerrit/mediawiki/vagrant/mediawiki$ git review -d 424609
...

~/Documents/gerrit/mediawiki/vagrant/mediawiki$ npm i
...

~/Documents/gerrit/mediawiki/vagrant/mediawiki$ npm run selenium

> @ selenium /Users/z/Documents/gerrit/mediawiki/vagrant/mediawiki
> killall -0 chromedriver 2>/dev/null || chromedriver --url-base=/wd/hub --port=4444 & wdio tests/selenium/wdio.conf.js; killall chromedriver

Starting ChromeDriver 2.37.544337 (8c0344a12e552148c185f7d5117db1f28d6c9e85) on port 4444
Only local connections are allowed.
GVA info: Successfully connected to the Intel plugin, offline Gen7 
ERROR: Cannot read property 'split' of undefined
chrome
Type    at orderTestCases (/Users/z/Documents/gerrit/mediawiki/vagrant/mediawiki/node_modules/cucumber/lib/cli/helpers.js:142:28)
    at Object.<anonymous> (/Users/z/Documents/gerrit/mediawiki/vagrant/mediawiki/node_modules/cucumber/lib/cli/helpers.js:63:5)
    at Generator.next (<anonymous>)
    at Generator.tryCatcher (/Users/z/Documents/gerrit/mediawiki/vagrant/mediawiki/node_modules/bluebird/js/release/util.js:16:23)
    at PromiseSpawn._promiseFulfilled (/Users/z/Documents/gerrit/mediawiki/vagrant/mediawiki/node_modules/bluebird/js/release/generators.js:97:49)
    at Promise._settlePromise (/Users/z/Documents/gerrit/mediawiki/vagrant/mediawiki/node_modules/bluebird/js/release/promise.js:574:26)
    at Promise._settlePromise0 (/Users/z/Documents/gerrit/mediawiki/vagrant/mediawiki/node_modules/bluebird/js/release/promise.js:614:10)
    at Promise._settlePromises (/Users/z/Documents/gerrit/mediawiki/vagrant/mediawiki/node_modules/bluebird/js/release/promise.js:693:18)
    at Async._drainQueue (/Users/z/Documents/gerrit/mediawiki/vagrant/mediawiki/node_modules/bluebird/js/release/async.js:133:16)
    at Async._drainQueues (/Users/z/Documents/gerrit/mediawiki/vagrant/mediawiki/node_modules/bluebird/js/release/async.js:143:10)
    at Immediate.Async.drainQueues [as _onImmediate] (/Users/z/Documents/gerrit/mediawiki/vagrant/mediawiki/node_modules/bluebird/js/release/async.js:17:14)
    at runCallback (timers.js:763:18)
    at tryOnImmediate (timers.js:734:5)
    at processImmediate (timers.js:716:5)
    at process.topLevelDomainCallback (domain.js:102:23)
------------------------------------------------------------------
[chrome #0-0] Session ID: 37ff299cc0f3c97718294da27986452c
[chrome #0-0] Spec: /Users/z/Documents/gerrit/mediawiki/vagrant/mediawiki/skins/MinervaNeue/tests/selenium/features/category.feature
[chrome #0-0] Running: chrome
[chrome #0-0]
[chrome #0-0] Categories
[chrome #0-0]
[chrome #0-0]     I can view categories
[chrome #0-0]
[chrome #0-0]
[chrome #0-0]

GVA info: Successfully connected to the Intel plugin, offline Gen7 
ERROR: Cannot read property 'split' of undefined
chrome
Type    at orderTestCases (/Users/z/Documents/gerrit/mediawiki/vagrant/mediawiki/node_modules/cucumber/lib/cli/helpers.js:142:28)
    at Object.<anonymous> (/Users/z/Documents/gerrit/mediawiki/vagrant/mediawiki/node_modules/cucumber/lib/cli/helpers.js:63:5)
    at Generator.next (<anonymous>)
    at Generator.tryCatcher (/Users/z/Documents/gerrit/mediawiki/vagrant/mediawiki/node_modules/bluebird/js/release/util.js:16:23)
    at PromiseSpawn._promiseFulfilled (/Users/z/Documents/gerrit/mediawiki/vagrant/mediawiki/node_modules/bluebird/js/release/generators.js:97:49)
    at Promise._settlePromise (/Users/z/Documents/gerrit/mediawiki/vagrant/mediawiki/node_modules/bluebird/js/release/promise.js:574:26)
    at Promise._settlePromise0 (/Users/z/Documents/gerrit/mediawiki/vagrant/mediawiki/node_modules/bluebird/js/release/promise.js:614:10)
    at Promise._settlePromises (/Users/z/Documents/gerrit/mediawiki/vagrant/mediawiki/node_modules/bluebird/js/release/promise.js:693:18)
    at Async._drainQueue (/Users/z/Documents/gerrit/mediawiki/vagrant/mediawiki/node_modules/bluebird/js/release/async.js:133:16)
    at Async._drainQueues (/Users/z/Documents/gerrit/mediawiki/vagrant/mediawiki/node_modules/bluebird/js/release/async.js:143:10)
    at Immediate.Async.drainQueues [as _onImmediate] (/Users/z/Documents/gerrit/mediawiki/vagrant/mediawiki/node_modules/bluebird/js/release/async.js:17:14)
    at runCallback (timers.js:763:18)
    at tryOnImmediate (timers.js:734:5)
    at processImmediate (timers.js:716:5)
    at process.topLevelDomainCallback (domain.js:102:23)
------------------------------------------------------------------
[chrome #0-1] Session ID: f84427c62d5919f8c7fb2e5f70faf6c6
[chrome #0-1] Spec: /Users/z/Documents/gerrit/mediawiki/vagrant/mediawiki/skins/MinervaNeue/tests/selenium/features/diff.feature
[chrome #0-1] Running: chrome
[chrome #0-1]
[chrome #0-1] Page diff
[chrome #0-1]
[chrome #0-1]     Added and removed content
[chrome #0-1]
[chrome #0-1]
[chrome #0-1]



==================================================================
Number of specs: 2
sh: line 1:  7904 Terminated: 15          chromedriver --url-base=/wd/hub --port=4444

No. That is the overall epic. This task is complete when we have one test ported over.

Thanks. Now that I have read the task name carefully, I can see it myself. :)

Change 425422 had a related patch set uploaded (by Jdrewniak; owner: Jdrewniak):
[mediawiki/skins/MinervaNeue@master] Adding "diff.feature" selenium test

https://gerrit.wikimedia.org/r/425422

@zeljkofilipin I'm looking into the "split" error. When I update the node dependancies in the Minerva patch, I get the same error. There was a major version release of cucumber.js (4.0) in January, and I think there might be errors in the minerva patch when running with this new version.

If you want to get the core patch running, changing this line in package.json should get it working

from: 
"wdio-cucumber-framework": "^2.0.1",
to: 
"wdio-cucumber-framework": "^1.1.1",

(But I'll update the Minerva patch to run the new version anyway)

Tests are now running from core! But, something is wrong:

TypeError: feature.getFailureException is not a function
at SauceService.afterStep (/Users/z/Documents/gerrit/mediawiki/vagrant/mediawiki/node_modules/wdio-sauce-service/build/sauce-service.js:87:25)
at execHook (/Users/z/Documents/gerrit/mediawiki/vagrant/mediawiki/node_modules/wdio-sync/build/index.js:166:35)

424609/11 removes sauce labs configuration and removes error messages! :D

Just for a quick update on this, it looks like we have a lot of the pieces in place here.

A couple of patches, 424592 and 426884, enable running the tests in CI (T179190), and have recently been merged.

Patch 423648 contains the initial tests and is ready for a final round of code-review.

Patch 424609 enables running the tests from mediawiki-core, and depends on the patch above.

Once these two patches have been merged, I'm not sure what else needs to happen (probably some CI stuff) but I think we're well on our way get this going :)

@zeljkofilipin thanks for all your work on this so far!

Once these two patches have been merged, I'm not sure what else needs to happen (probably some CI stuff) but I think we're well on our way get this going :)

CI was changed with the introduction of Quibble (see Introducing Quibble, a test runner for MediaWiki). I am looking into how to run Cucumber using Quibble.

Hey @zeljkofilipin we were wondering about this task today. What's the current status/blockers and can we help with anything?

Sorry, I was busy with Quibble (Blog Post: Run Selenium tests using Quibble and Docker) and reviewing many Selenium framework related patches created by @Krinkle.

My plan for the Barcelona hackathon is to spend about 80% of the time pairing with people (T190687: Pair on writing Selenium tests in JavaScript/Node.js). Would you like to pair on this?

In the meantime, I will continue working on this.

Any updates on this @zeljkofilipin we're probably going to have to drop this from our board next week and won't have bandwidth to work on it for another month. I am working European hours this week if pairing with @Jdrewniak or I would help unblock this.

Jdlrobson changed the task status from Open to Stalled.Jun 4 2018, 5:31 PM

Moving out of the sprint. We're unable to work on this right now. As soon as the CI work is resolved, we'll take a look at rescheduling this work.

Ragesoss updated the task description. (Show Details)Jun 6 2018, 10:36 PM

Any updates on this @zeljkofilipin we're probably going to have to drop this from our board next week and won't have bandwidth to work on it for another month. I am working European hours this week if pairing with @Jdrewniak or I would help unblock this.

Sorry, I was busy with other tasks. ๐Ÿ˜ข I will get back to this as soon as possible.

zeljkofilipin added a subscriber: Jdrewniak.
zeljkofilipin removed zeljkofilipin as the assignee of this task.Jun 28 2018, 5:11 PM

Sorry on being slow with this, I am busy with other things. This is high on my todo list, but I am not working on it at the moment.

zeljkofilipin changed the task status from Stalled to Open.Jul 6 2018, 4:43 PM
zeljkofilipin claimed this task.
zeljkofilipin updated the task description. (Show Details)Jul 6 2018, 4:52 PM
zeljkofilipin updated the task description. (Show Details)Jul 6 2018, 4:58 PM
zeljkofilipin updated the task description. (Show Details)
zeljkofilipin changed the status of subtask T179190: Run Selenium Cucumber tests in CI from Open to Stalled.Jul 17 2018, 8:57 AM
Jdlrobson changed the task status from Stalled to Open.Apr 5 2019, 4:46 PM

Re-opening for visibility as this is know causing my team huge problems in that we've had to disable a large amount of tests that are no longer passing. What is the bare minimum we need to do to get a Ruby and Node.js browser tests running in Minerva side by side? Is T199116 really a blocker or if we removed the cucumber part of https://gerrit.wikimedia.org/r/423648 would that suffice?

Change 423648 had a related patch set uploaded (by Jdlrobson; owner: Jdrewniak):
[mediawiki/skins/MinervaNeue@master] Adding initial webdriver.io test

https://gerrit.wikimedia.org/r/423648

@Krinkle in IRC felt this might be a 1 line config change? Is that true? If so could we enable those now? I will probably have some time this afternoon to revise Adding initial webdriver.io test to not use Cucumber given T199116 is blocked.

Krinkle added a comment.EditedApr 5 2019, 4:55 PM

Yep, I'll take a look after T220217 is fixed in prod.

Re-opening for visibility as this is know causing my team huge problems in that we've had to disable a large amount of tests that are no longer passing. What is the bare minimum we need to do to get a Ruby and Node.js browser tests running in Minerva side by side? Is T199116 really a blocker or if we removed the cucumber part of https://gerrit.wikimedia.org/r/423648 would that suffice?

Yes. CI currently supports Mocha. If you wrote tests in Mocha instead of Cucumber, they would just work.

Krinkle https://gerrit.wikimedia.org/r/#/c/mediawiki/skins/MinervaNeue/+/423648/ is now using Mocha. Let's see if we can get this running in CI and a new job setup and listed on https://integration.wikimedia.org/ci/view/Reading-Web/job/selenium-MinervaNeue/ - if so happy to talk to my team about removing the Cucumber dependency.

Change 423648 merged by jenkins-bot:
[mediawiki/skins/MinervaNeue@master] Adding initial webdriver.io test

https://gerrit.wikimedia.org/r/423648

This comment was removed by Krinkle.

Ah, right. Quibble jobs auto-detect the wdio specs and run them. I was thinking we need to add a line to layout.yaml in ci-config to enable that. Even better!

Change 502625 had a related patch set uploaded (by Jdlrobson; owner: Jdlrobson):
[mediawiki/skins/MinervaNeue@master] WIP: Port remaining @login tests to Node.js

https://gerrit.wikimedia.org/r/502625

Change 502884 had a related patch set uploaded (by Jdlrobson; owner: Jdlrobson):
[integration/config@master] Run Minerva browser tests against the mobile domain

https://gerrit.wikimedia.org/r/502884

Change 503033 had a related patch set uploaded (by Jdlrobson; owner: Jdlrobson):
[mediawiki/skins/MinervaNeue@master] Run Minerva browser tests against the mobile domain

https://gerrit.wikimedia.org/r/503033

Change 502884 abandoned by Jdlrobson:
Run Minerva browser tests against the mobile domain

Reason:
Thanks for the helpful pointer Hashar!
https://gerrit.wikimedia.org/r/#/c/mediawiki/skins/MinervaNeue/ /503033 Run Minerva browser tests against the mobile domain

https://gerrit.wikimedia.org/r/502884

Change 503033 merged by jenkins-bot:
[mediawiki/skins/MinervaNeue@master] Run Minerva browser tests against the mobile domain

https://gerrit.wikimedia.org/r/503033

Jdlrobson removed a project: Patch-For-Review.

This is done now. Minerva is running both Ruby and Node.js browser tests side by side against beta cluster and against every commit.

Jdlrobson closed this task as Resolved.Apr 11 2019, 5:57 PM
Jdlrobson claimed this task.
Jdlrobson updated the task description. (Show Details)