In T190710#4099936, @Jdrewniak wrote:

I don't know if the Selenium CI jobs are set up use testing frameworks other than mocha. In the patch had to extend the mediawiki-core wdio.conf.js file to specify 'cucumber' as the testing framework, but I don't think CI picks up that file.

At the moment, CI only runs Mocha tests.

In T190710#4101634, @Jdlrobson wrote:

CI is wired up to scan a specs folder for JS files but /home/jenkins/workspace/mediawiki-core-qunit-selenium-jessie/src/skins/*/tests/selenium/specs/**/*.js did not match any file

I'll take a look at how to make the job run both Mocha and Cucumber. Webdriverio support both, but I do not think I have ever tried to run both at the same time. There should be no technical difficulties. In Ruby world it was not unusual that a project has both RSpec and Cucumber tests, so I guess it is similar in Node.js with Mocha and Cucumber.

We're using features here so CI will needed to be able to deal with this different setup (which means loading Minerva's config). Since CirrusSearch is also using this, this would be helpful so they can stop using Cindy and rely on Jenkins CI.

CirrusSearch is a different story. It's not Cucumber that makes it difficult to run Selenium test, it's because CI at the moment does not support CirrusSearch. :)

Zeljko says this should be doable. He's going to take a look at it tomorrow and report back. He will do it in core so other extensions can make use of this - doing this in MobileFrontend should be pretty straightforward. Yay!

Update: webdriverio does not support running both mocha and cucumber at the same time. I am working on a solution to run them separately. See T179190 for details. I estimate to have something working tomorrow (Saturday).

Does this task replace T174018: Port Minerva's 7 Ruby-based browser tests to Selenium Node.js?

Apologies for the delay, this proved to be a bit more complicated than I thought. 🤔 I have created a few patches while working on this, see topic:T179190. Reviews are welcome.

TLDR I can run Minerva Cucumber tests from Minerva folder, but not from core. 😕

Provisioning:

~/Documents/gerrit/mediawiki/vagrant$ vagrant up
...

~/Documents/gerrit/mediawiki/vagrant$ vagrant roles enable minerva
Ok. Run `vagrant provision` to apply your changes.

~/Documents/gerrit/mediawiki/vagrant$ vagrant provision
...

Running tests from Minerva:

~/Documents/gerrit/mediawiki/vagrant$ cd mediawiki/skins/MinervaNeue/

~/Documents/gerrit/mediawiki/vagrant/mediawiki/skins/MinervaNeue$ git review -d 423648
...

~/Documents/gerrit/mediawiki/vagrant/mediawiki/skins/MinervaNeue$ npm i
...

~/Documents/gerrit/mediawiki/vagrant/mediawiki/skins/MinervaNeue$ npm run selenium
...
Number of specs: 2
13 passing (51.20s)
sh: line 1:  7831 Terminated: 15          chromedriver --url-base=/wd/hub --port=4444

So far so good.

In T190710#4120736, @zeljkofilipin wrote:

Does this task replace T174018: Port Minerva's 7 Ruby-based browser tests to Selenium Node.js?

No. That is the overall epic. This task is complete when we have one test ported over.

I am doing something wrong in the core patch (424609). I am not sure where ERROR: Cannot read property 'split' of undefined comes from:

~/Documents/gerrit/mediawiki/vagrant/mediawiki/skins/MinervaNeue$ cd ../..

~/Documents/gerrit/mediawiki/vagrant/mediawiki$ git review -d 424609
...

~/Documents/gerrit/mediawiki/vagrant/mediawiki$ npm i
...

~/Documents/gerrit/mediawiki/vagrant/mediawiki$ npm run selenium

> @ selenium /Users/z/Documents/gerrit/mediawiki/vagrant/mediawiki
> killall -0 chromedriver 2>/dev/null || chromedriver --url-base=/wd/hub --port=4444 & wdio tests/selenium/wdio.conf.js; killall chromedriver

Starting ChromeDriver 2.37.544337 (8c0344a12e552148c185f7d5117db1f28d6c9e85) on port 4444
Only local connections are allowed.
GVA info: Successfully connected to the Intel plugin, offline Gen7 
ERROR: Cannot read property 'split' of undefined
chrome
Type    at orderTestCases (/Users/z/Documents/gerrit/mediawiki/vagrant/mediawiki/node_modules/cucumber/lib/cli/helpers.js:142:28)
    at Object.<anonymous> (/Users/z/Documents/gerrit/mediawiki/vagrant/mediawiki/node_modules/cucumber/lib/cli/helpers.js:63:5)
    at Generator.next (<anonymous>)
    at Generator.tryCatcher (/Users/z/Documents/gerrit/mediawiki/vagrant/mediawiki/node_modules/bluebird/js/release/util.js:16:23)
    at PromiseSpawn._promiseFulfilled (/Users/z/Documents/gerrit/mediawiki/vagrant/mediawiki/node_modules/bluebird/js/release/generators.js:97:49)
    at Promise._settlePromise (/Users/z/Documents/gerrit/mediawiki/vagrant/mediawiki/node_modules/bluebird/js/release/promise.js:574:26)
    at Promise._settlePromise0 (/Users/z/Documents/gerrit/mediawiki/vagrant/mediawiki/node_modules/bluebird/js/release/promise.js:614:10)
    at Promise._settlePromises (/Users/z/Documents/gerrit/mediawiki/vagrant/mediawiki/node_modules/bluebird/js/release/promise.js:693:18)
    at Async._drainQueue (/Users/z/Documents/gerrit/mediawiki/vagrant/mediawiki/node_modules/bluebird/js/release/async.js:133:16)
    at Async._drainQueues (/Users/z/Documents/gerrit/mediawiki/vagrant/mediawiki/node_modules/bluebird/js/release/async.js:143:10)
    at Immediate.Async.drainQueues [as _onImmediate] (/Users/z/Documents/gerrit/mediawiki/vagrant/mediawiki/node_modules/bluebird/js/release/async.js:17:14)
    at runCallback (timers.js:763:18)
    at tryOnImmediate (timers.js:734:5)
    at processImmediate (timers.js:716:5)
    at process.topLevelDomainCallback (domain.js:102:23)
------------------------------------------------------------------
[chrome #0-0] Session ID: 37ff299cc0f3c97718294da27986452c
[chrome #0-0] Spec: /Users/z/Documents/gerrit/mediawiki/vagrant/mediawiki/skins/MinervaNeue/tests/selenium/features/category.feature
[chrome #0-0] Running: chrome
[chrome #0-0]
[chrome #0-0] Categories
[chrome #0-0]
[chrome #0-0]     I can view categories
[chrome #0-0]
[chrome #0-0]
[chrome #0-0]

GVA info: Successfully connected to the Intel plugin, offline Gen7 
ERROR: Cannot read property 'split' of undefined
chrome
Type    at orderTestCases (/Users/z/Documents/gerrit/mediawiki/vagrant/mediawiki/node_modules/cucumber/lib/cli/helpers.js:142:28)
    at Object.<anonymous> (/Users/z/Documents/gerrit/mediawiki/vagrant/mediawiki/node_modules/cucumber/lib/cli/helpers.js:63:5)
    at Generator.next (<anonymous>)
    at Generator.tryCatcher (/Users/z/Documents/gerrit/mediawiki/vagrant/mediawiki/node_modules/bluebird/js/release/util.js:16:23)
    at PromiseSpawn._promiseFulfilled (/Users/z/Documents/gerrit/mediawiki/vagrant/mediawiki/node_modules/bluebird/js/release/generators.js:97:49)
    at Promise._settlePromise (/Users/z/Documents/gerrit/mediawiki/vagrant/mediawiki/node_modules/bluebird/js/release/promise.js:574:26)
    at Promise._settlePromise0 (/Users/z/Documents/gerrit/mediawiki/vagrant/mediawiki/node_modules/bluebird/js/release/promise.js:614:10)
    at Promise._settlePromises (/Users/z/Documents/gerrit/mediawiki/vagrant/mediawiki/node_modules/bluebird/js/release/promise.js:693:18)
    at Async._drainQueue (/Users/z/Documents/gerrit/mediawiki/vagrant/mediawiki/node_modules/bluebird/js/release/async.js:133:16)
    at Async._drainQueues (/Users/z/Documents/gerrit/mediawiki/vagrant/mediawiki/node_modules/bluebird/js/release/async.js:143:10)
    at Immediate.Async.drainQueues [as _onImmediate] (/Users/z/Documents/gerrit/mediawiki/vagrant/mediawiki/node_modules/bluebird/js/release/async.js:17:14)
    at runCallback (timers.js:763:18)
    at tryOnImmediate (timers.js:734:5)
    at processImmediate (timers.js:716:5)
    at process.topLevelDomainCallback (domain.js:102:23)
------------------------------------------------------------------
[chrome #0-1] Session ID: f84427c62d5919f8c7fb2e5f70faf6c6
[chrome #0-1] Spec: /Users/z/Documents/gerrit/mediawiki/vagrant/mediawiki/skins/MinervaNeue/tests/selenium/features/diff.feature
[chrome #0-1] Running: chrome
[chrome #0-1]
[chrome #0-1] Page diff
[chrome #0-1]
[chrome #0-1]     Added and removed content
[chrome #0-1]
[chrome #0-1]
[chrome #0-1]



==================================================================
Number of specs: 2
sh: line 1:  7904 Terminated: 15          chromedriver --url-base=/wd/hub --port=4444

In T190710#4120828, @Jdlrobson wrote:

No. That is the overall epic. This task is complete when we have one test ported over.

Thanks. Now that I have read the task name carefully, I can see it myself. :)

Change 425422 had a related patch set uploaded (by Jdrewniak; owner: Jdrewniak):
[mediawiki/skins/MinervaNeue@master] Adding "diff.feature" selenium test

https://gerrit.wikimedia.org/r/425422

@zeljkofilipin I'm looking into the "split" error. When I update the node dependancies in the Minerva patch, I get the same error. There was a major version release of cucumber.js (4.0) in January, and I think there might be errors in the minerva patch when running with this new version.

If you want to get the core patch running, changing this line in package.json should get it working

from: 
"wdio-cucumber-framework": "^2.0.1",
to: 
"wdio-cucumber-framework": "^1.1.1",

(But I'll update the Minerva patch to run the new version anyway)

Tests are now running from core! But, something is wrong:

TypeError: feature.getFailureException is not a function
at SauceService.afterStep (/Users/z/Documents/gerrit/mediawiki/vagrant/mediawiki/node_modules/wdio-sauce-service/build/sauce-service.js:87:25)
at execHook (/Users/z/Documents/gerrit/mediawiki/vagrant/mediawiki/node_modules/wdio-sync/build/index.js:166:35)

424609/11 removes sauce labs configuration and removes error messages! :D

Just for a quick update on this, it looks like we have a lot of the pieces in place here.

A couple of patches, 424592 and 426884, enable running the tests in CI (T179190), and have recently been merged.

Patch 423648 contains the initial tests and is ready for a final round of code-review.

Patch 424609 enables running the tests from mediawiki-core, and depends on the patch above.

Once these two patches have been merged, I'm not sure what else needs to happen (probably some CI stuff) but I think we're well on our way get this going :)

@zeljkofilipin thanks for all your work on this so far!

In T190710#4153110, @Jdrewniak wrote:

Once these two patches have been merged, I'm not sure what else needs to happen (probably some CI stuff) but I think we're well on our way get this going :)

CI was changed with the introduction of Quibble (see Introducing Quibble, a test runner for MediaWiki). I am looking into how to run Cucumber using Quibble.

Hey @zeljkofilipin we were wondering about this task today. What's the current status/blockers and can we help with anything?

Sorry, I was busy with Quibble (Blog Post: Run Selenium tests using Quibble and Docker) and reviewing many Selenium framework related patches created by @Krinkle.

My plan for the Barcelona hackathon is to spend about 80% of the time pairing with people (T190687: Pair on writing Selenium tests in JavaScript/Node.js). Would you like to pair on this?

In the meantime, I will continue working on this.

Any updates on this @zeljkofilipin we're probably going to have to drop this from our board next week and won't have bandwidth to work on it for another month. I am working European hours this week if pairing with @Jdrewniak or I would help unblock this.

In T190710#4240299, @Jdlrobson wrote:

Any updates on this @zeljkofilipin we're probably going to have to drop this from our board next week and won't have bandwidth to work on it for another month. I am working European hours this week if pairing with @Jdrewniak or I would help unblock this.

Sorry, I was busy with other tasks. 😢 I will get back to this as soon as possible.

Change 423648 had a related patch set uploaded (by Jdlrobson; owner: Jdrewniak):
[mediawiki/skins/MinervaNeue@master] Adding initial webdriver.io test

https://gerrit.wikimedia.org/r/423648

@Krinkle in IRC felt this might be a 1 line config change? Is that true? If so could we enable those now? I will probably have some time this afternoon to revise Adding initial webdriver.io test to not use Cucumber given T199116 is blocked.

Yep, I'll take a look after T220217 is fixed in prod.

In T190710#5088918, @Jdlrobson wrote:

Re-opening for visibility as this is know causing my team huge problems in that we've had to disable a large amount of tests that are no longer passing. What is the bare minimum we need to do to get a Ruby and Node.js browser tests running in Minerva side by side? Is T199116 really a blocker or if we removed the cucumber part of https://gerrit.wikimedia.org/r/423648 would that suffice?

Yes. CI currently supports Mocha. If you wrote tests in Mocha instead of Cucumber, they would just work.

Krinkle https://gerrit.wikimedia.org/r/#/c/mediawiki/skins/MinervaNeue/+/423648/ is now using Mocha. Let's see if we can get this running in CI and a new job setup and listed on https://integration.wikimedia.org/ci/view/Reading-Web/job/selenium-MinervaNeue/ - if so happy to talk to my team about removing the Cucumber dependency.

Change 423648 merged by jenkins-bot:
[mediawiki/skins/MinervaNeue@master] Adding initial webdriver.io test

https://gerrit.wikimedia.org/r/423648

Ah, right. Quibble jobs auto-detect the wdio specs and run them. I was thinking we need to add a line to layout.yaml in ci-config to enable that. Even better!

Change 502625 had a related patch set uploaded (by Jdlrobson; owner: Jdlrobson):
[mediawiki/skins/MinervaNeue@master] WIP: Port remaining @login tests to Node.js

https://gerrit.wikimedia.org/r/502625

Blocked on RelEng (T220658) - we have nobody with +2 rights in that repo.

Change 502884 had a related patch set uploaded (by Jdlrobson; owner: Jdlrobson):
[integration/config@master] Run Minerva browser tests against the mobile domain

https://gerrit.wikimedia.org/r/502884

Change 503033 had a related patch set uploaded (by Jdlrobson; owner: Jdlrobson):
[mediawiki/skins/MinervaNeue@master] Run Minerva browser tests against the mobile domain

https://gerrit.wikimedia.org/r/503033

Change 502884 abandoned by Jdlrobson:
Run Minerva browser tests against the mobile domain

Reason:
Thanks for the helpful pointer Hashar!
https://gerrit.wikimedia.org/r/#/c/mediawiki/skins/MinervaNeue/ /503033 Run Minerva browser tests against the mobile domain

https://gerrit.wikimedia.org/r/502884

Change 503033 merged by jenkins-bot:
[mediawiki/skins/MinervaNeue@master] Run Minerva browser tests against the mobile domain

https://gerrit.wikimedia.org/r/503033

This is done now. Minerva is running both Ruby and Node.js browser tests side by side against beta cluster and against every commit.