Page MenuHomePhabricator

Run Selenium Cucumber tests in CI
Open, Stalled, HighPublic

Description

Status

Blocked on T199116: Quibble should run `npm install` and `npm run selenium-test` for each extension/skin that has Selenium tests.

Description

Needed for T190710: Minerva Ruby and Node.js browser tests running side by side. The initial idea was to run both core and extension/skin tests from core, using either mocha or cucumber (or possibly both). I do not think that is a good idea any more. Looks like the way forward is to run tests for each repository (core, extension, skin) from the repository.

TODO

๐Ÿš€ T199116: Quibble should run `npm install` and `npm run selenium-test` for each extension/skin that has Selenium tests

Merged

๐Ÿ 428040 Run npm run selenium-test instead of grunt webdriver:test
๐Ÿ 424567 Selenium: use default value for waitforTimeout
๐Ÿ 424592 Run npm run selenium instead of grunt webdriver:test
๐Ÿ 424763 selenium: Remove Jenkins configuration file
๐Ÿ 427362 Selenium: Jenkins configuration file no longer needed

๐Ÿ 424764 Selenium: move logic for running tests from package.json to selenium.sh
๐Ÿ 426881 Selenium: move logic for running tests from package.json to selenium.sh
๐Ÿ 426883 Selenium: move logic for running tests from package.json to selenium.sh
๐Ÿ 426884 Selenium: move logic for running tests from package.json to selenium.sh

๐Ÿ 429018 selenium: Abstract test command behind 'npm run'
๐Ÿ 428344 selenium: Abstract test command behind 'npm run'
๐Ÿ 428373 selenium: Abstract test command behind 'npm run'
๐Ÿ 428374 selenium: Abstract test command behind 'npm run'
๐Ÿ 428375 selenium: Abstract test command behind 'npm run'

Abandoned

โšฐ๏ธ 391546 WIP Run Cucumber+Selenium+Node.js in CI
โšฐ๏ธ 391547 WIP Run Cucumber+Selenium+Node.js in CI
โšฐ๏ธ 391566 WIP Run Cucumber+Selenium+Node.js in CI
โšฐ๏ธ 424609 WIP Selenium: run both Mocha and Cucumber tests

Related Objects

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
zeljkofilipin raised the priority of this task from Low to High.Apr 6 2018, 11:01 AM
zeljkofilipin updated the task description. (Show Details)

Change 424567 had a related patch set uploaded (by Zfilipin; owner: Zfilipin):
[mediawiki/core@master] Selenium: use default value for waitforTimeout

https://gerrit.wikimedia.org/r/424567

zeljkofilipin updated the task description. (Show Details)Apr 6 2018, 1:22 PM
zeljkofilipin updated the task description. (Show Details)

Change 424592 had a related patch set uploaded (by Zfilipin; owner: Zfilipin):
[integration/config@master] WIP Run Cucumber+Selenium+Node.js in CI

https://gerrit.wikimedia.org/r/424592

Change 424609 had a related patch set uploaded (by Zfilipin; owner: Zfilipin):
[mediawiki/core@master] WIP Selenium: run Cucumber tests

https://gerrit.wikimedia.org/r/424609

zeljkofilipin renamed this task from Run Cucumber+Selenium+Node.js in CI to Run Selenium Cucumber tests in CI.Apr 6 2018, 2:46 PM

Change 424764 had a related patch set uploaded (by Zfilipin; owner: Zfilipin):
[mediawiki/core@master] Selenium: cun npm run selenium instead of grunt webdriver:test in CI

https://gerrit.wikimedia.org/r/424764

Change 424763 had a related patch set uploaded (by Zfilipin; owner: Zfilipin):
[mediawiki/core@master] WIP Selenium: remove Jenkins configuration file

https://gerrit.wikimedia.org/r/424763

Change 424567 merged by jenkins-bot:
[mediawiki/core@master] Selenium: use default value for waitforTimeout

https://gerrit.wikimedia.org/r/424567

Tests are now running from core! But, something is wrong:

TypeError: feature.getFailureException is not a function
at SauceService.afterStep (/Users/z/Documents/gerrit/mediawiki/vagrant/mediawiki/node_modules/wdio-sauce-service/build/sauce-service.js:87:25)
at execHook (/Users/z/Documents/gerrit/mediawiki/vagrant/mediawiki/node_modules/wdio-sync/build/index.js:166:35)

424609/11 removes sauce labs configuration and removes error messages! :D

Change 424764 merged by Krinkle:
[mediawiki/core@master] Selenium: move logic for running tests from package.json to selenium.sh

https://gerrit.wikimedia.org/r/424764

Change 424763 merged by Krinkle:
[mediawiki/core@master] selenium: Remove Jenkins configuration file

https://gerrit.wikimedia.org/r/424763

Change 426881 had a related patch set uploaded (by Zfilipin; owner: Zfilipin):
[mediawiki/core@REL1_30] Selenium: move logic for running tests from package.json to selenium.sh

https://gerrit.wikimedia.org/r/426881

Change 426883 had a related patch set uploaded (by Zfilipin; owner: Zfilipin):
[mediawiki/core@REL1_29] Selenium: move logic for running tests from package.json to selenium.sh

https://gerrit.wikimedia.org/r/426883

Change 426884 had a related patch set uploaded (by Zfilipin; owner: Zfilipin):
[mediawiki/core@wmf/1.31.0-wmf.29] Selenium: move logic for running tests from package.json to selenium.sh

https://gerrit.wikimedia.org/r/426884

Change 427362 had a related patch set uploaded (by Zfilipin; owner: Zfilipin):
[integration/config@master] Selenium: Jenkins configuration file no longer needed

https://gerrit.wikimedia.org/r/427362

Change 426881 merged by jenkins-bot:
[mediawiki/core@REL1_30] Selenium: move logic for running tests from package.json to selenium.sh

https://gerrit.wikimedia.org/r/426881

Change 426883 merged by jenkins-bot:
[mediawiki/core@REL1_29] Selenium: move logic for running tests from package.json to selenium.sh

https://gerrit.wikimedia.org/r/426883

Change 427362 merged by jenkins-bot:
[integration/config@master] Selenium: Jenkins configuration file no longer needed

https://gerrit.wikimedia.org/r/427362

Change 428040 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[integration/quibble@master] Run npm run selenium instead of grunt webdriver:test

https://gerrit.wikimedia.org/r/428040

Change 424592 merged by Krinkle:
[integration/config@master] Run npm run selenium instead of grunt webdriver:test

https://gerrit.wikimedia.org/r/424592

Change 426884 merged by jenkins-bot:
[mediawiki/core@wmf/1.31.0-wmf.29] Selenium: move logic for running tests from package.json to selenium.sh

https://gerrit.wikimedia.org/r/426884

Change 428344 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[mediawiki/core@master] selenium: Abstract test command behind 'npm run'

https://gerrit.wikimedia.org/r/428344

Change 428373 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[mediawiki/core@REL1_31] selenium: Abstract test command behind 'npm run'

https://gerrit.wikimedia.org/r/428373

Change 428374 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[mediawiki/core@REL1_30] selenium: Abstract test command behind 'npm run'

https://gerrit.wikimedia.org/r/428374

Change 428375 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[mediawiki/core@REL1_29] selenium: Abstract test command behind 'npm run'

https://gerrit.wikimedia.org/r/428375

Change 428344 merged by jenkins-bot:
[mediawiki/core@master] selenium: Abstract test command behind 'npm run'

https://gerrit.wikimedia.org/r/428344

Change 428375 merged by jenkins-bot:
[mediawiki/core@REL1_29] selenium: Abstract test command behind 'npm run'

https://gerrit.wikimedia.org/r/428375

Change 428374 merged by jenkins-bot:
[mediawiki/core@REL1_30] selenium: Abstract test command behind 'npm run'

https://gerrit.wikimedia.org/r/428374

Change 428373 merged by jenkins-bot:
[mediawiki/core@REL1_31] selenium: Abstract test command behind 'npm run'

https://gerrit.wikimedia.org/r/428373

Change 429018 had a related patch set uploaded (by Hashar; owner: Krinkle):
[mediawiki/core@wmf/1.31.0-wmf.30] selenium: Abstract test command behind 'npm run'

https://gerrit.wikimedia.org/r/429018

Change 428040 merged by jenkins-bot:
[integration/quibble@master] Run npm run selenium-test instead of grunt webdriver:test

https://gerrit.wikimedia.org/r/428040

Change 429018 merged by jenkins-bot:
[mediawiki/core@wmf/1.31.0-wmf.30] selenium: Abstract test command behind 'npm run'

https://gerrit.wikimedia.org/r/429018

Krinkle added a subscriber: Krinkle.May 2 2018, 2:35 AM
mediawiki/core:/package.json
+     "chai": "4.1.2",

@zeljkofilipin In the interest of not requiring as intense maintenance and up-keep for this new stack, I would recommend focussing on making the selenium test run from the repository rather than from core.

The above diff in particular stood out, because it reminded me of when we first introduced JSLint in Wikimedia's CI (back in 2012 I think?). We globally installed a version of JSHint and started running it on different repositories. All was well. It supported repo-local override settings as well. Then, it came time to upgrade to a newer version of JSHint and it proved extremely disruptive because the version was in one place only. The only right way to upgrade that "one" version would've been to somehow atomically update all repositories to conform to new rules. This is a problem because we don't maintain all repositories ourselves. And even the repositories we do maintain, you don't want to force a single point in time for one person (nor everyone together) to have to make an upgrade work.

We solved this by instead using the npm test entry point where repositories decide individually which version of what tool to use.

I suspect the idea to run from core may've been inspired by how PHPUnit tests are run, but I think the Selenium test stack is in a unique position to do it better.

Especially because you've already designed and implemented in a way that is (almost) independently runnable. The framework comes from npm. The configuration is stateless from CLI or environment variables. The only thing that (most) Selenium tests need from core is the page objects, which per T193088#4165721 could easily be published and pulled in locally.

I haven't spoken up before now (aside from T193088) because it was just a handful of files, which are already pulled in automatically by Jenkins, and are actually versions correctly for older branches thanks to Zuul. It seemed pretty minimal and easy to fix later. But now when adding Chai and Cucumber and other abstractions to it, it starts to become harder to fix later on. Let me know if you're interested in trying to get this working over the next week, so that the Cucumber addition can make use of it from the start.

Let me know if you're interested in trying to get this working over the next week, so that the Cucumber addition can make use of it from the start.

My plan for the Barcelona hackathon is to spend about 80% of the time pairing with people (T190687: Pair on writing Selenium tests in JavaScript/Node.js). Would you like to pair on this?

In the meantime, I will try to review (and merge) all existing commits.

zeljkofilipin removed zeljkofilipin as the assignee of this task.Jun 28 2018, 5:06 PM

@zeljkofilipin In the interest of not requiring as intense maintenance and up-keep for this new stack, I would recommend focussing on making the selenium test run from the repository rather than from core.

Agreed. I have talked with @hashar about it, and I think he agrees too. ๐Ÿ˜…

zeljkofilipin updated the task description. (Show Details)Jul 6 2018, 4:22 PM
zeljkofilipin updated the task description. (Show Details)Jul 6 2018, 4:26 PM
zeljkofilipin updated the task description. (Show Details)
zeljkofilipin updated the task description. (Show Details)Jul 6 2018, 4:30 PM
zeljkofilipin updated the task description. (Show Details)Jul 6 2018, 4:36 PM
zeljkofilipin updated the task description. (Show Details)

Change 424609 abandoned by Zfilipin:
WIP Selenium: run both Mocha and Cucumber tests

Reason:
Tests for a repo will run from the repo, not from core.

https://gerrit.wikimedia.org/r/424609

zeljkofilipin updated the task description. (Show Details)
zeljkofilipin removed a project: Patch-For-Review.
zeljkofilipin removed zeljkofilipin as the assignee of this task.Jul 17 2018, 9:04 AM
zeljkofilipin updated the task description. (Show Details)