Page MenuHomePhabricator

[Epic] Port Selenium tests from Ruby to Node.js for the Search Platform
Closed, ResolvedPublic

Description

See parent task T139740: Port Selenium tests from Ruby to Node.js for context.

This epic groups the work of migrating the selenium tests for CirrusSearch. The Search platform's Ruby tests live in a single directory (tests/browser) and we'll need to re-write all those tests that exercise the full stack by making requests to MediaWiki and inspecting the results; mostly by creating/updating pages and searching for them.

The test conversion will be part of the Q2 goals for the Search Platform team:

  • the team will take on tests as they see fit, as they have time to do so
  • most of the tests average about 8 -10 minutes to run (if they take longer, reach out to David C for guidance)

Where / What

  • CirrusSearch
  • tests running on cindy
  • probable hacks using ruby monkeypatching
  • ⬆️ Add any missing projects

Basic Process

  • Create a new directory for these new tests:
    • new name: integration
    • this step will only be done once!
  • view the list of tests:
  • copy the original test over to the new directory (so we don't lose the old/original test and removing one might cause issues for other tests)
    • After a few tests have been converted and the steps have been changed, it should make it easier for later tests
    • after the re-write, be sure to run the test to be sure nothing fails
  • once all tests have been converted, we'll then remove the older Selenium jobs in /features

The deadline for this is by end of October 2017. Currently, only critical problems are being resolved in the Ruby stack; after October, RelEng will no longer maintain the Ruby stack.

Details

Related Gerrit Patches:
mediawiki/extensions/CirrusSearch : masterRemove Ruby browser tests
integration/config : masterDo not run Ruby Jenkins jobs for CirrusSearch
integration/config : masterRun mediawiki-core-qunit-selenium-jessie for CirrusSearch

Event Timeline

debt created this task.Aug 24 2017, 9:22 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 24 2017, 9:22 PM
debt updated the task description. (Show Details)Aug 24 2017, 9:25 PM
Smalyshev updated the task description. (Show Details)Aug 24 2017, 11:27 PM
Restricted Application added a subscriber: jeblad. · View Herald TranscriptAug 25 2017, 8:58 AM
debt added a comment.Aug 29 2017, 5:24 PM

We'll start the porting over of tests when we run across them, simply because we don't generally go back to older tests until we need them again. We're unsure how much work it'll be to do, overall, until we actually start working on them, but the good news is that we're not blocking any other teams.

Cindy will run all tests on every commit, we might want to start there to figure out which tests we'll need to port over first. The tests don't need to be all converted at once, there can be a mixture of Ruby and Node tests within Cindy's processes (depending on state and other dependancies).

We'll have a meeting to decide what needs to be done and who will tackle it soon.

debt added a comment.Sep 6 2017, 5:22 PM

Rough notes from our meeting on 6 Sep 2017 in regards to Ruby to Node.js Conversion of Selenium tests on Cindy:

Cindy to Jenkins

do our tests run against the beta cluster?

deadline is end of oct 2017 - soft deadline, since we run all our own tests

Cucumber would be a layer on top of a basic coded test

writing the tests

  • probably don't need a lot of documentation with writing the tests in cucumber
  • need to get all the pre-conditions to work first
  • keeps feature files
  • with existing tags and hooks and world objects
debt updated the task description. (Show Details)Sep 6 2017, 5:38 PM
debt added subscribers: dcausse, TJones, Jdrewniak.

Cucumber-JS implementation: https://github.com/cucumber/cucumber-js
Since it's an official port, I'd expect most of the infrastructure to work, but we'd have to port Wikimedia Ruby parts of course.

debt added a comment.Sep 19 2017, 4:17 PM

Good information all written up by @Jdrewniak here: https://gerrit.wikimedia.org/r/#/c/378688/

[WIP] Ruby to Node conversion of integration tests.
Begininng the conversion of integration tests from Ruby to Node.
Maintaining Cucumber (here Cucumber.js) as the testing framework.
https://github.com/cucumber/cucumber-js
Using MWBot as the API helper
https://github.com/Fannon/mwbot
Requires chromedriver to be installed:
https://sites.google.com/a/chromium.org/chromedriver/
Currently set to run in Chrome (not yet headless).

debt updated the task description. (Show Details)Sep 19 2017, 4:18 PM

I will create a simple test following instructions at Selenium/Node.js/Write. I will also set up CI to run the tests.

Change 381785 had a related patch set uploaded (by Zfilipin; owner: Zfilipin):
[mediawiki/extensions/CirrusSearch@master] WIP The first Selenium test in Node.js

https://gerrit.wikimedia.org/r/381785

I should have tagged this on a few patches, we actually have a small, but reasonable, amount of work done on this: https://gerrit.wikimedia.org/r/#/c/381513/

Looks like this is already happening, for example: https://gerrit.wikimedia.org/r/#/c/381513/

Change 381791 had a related patch set uploaded (by Zfilipin; owner: Zfilipin):
[integration/config@master] Run mediawiki-core-qunit-selenium-jessie for CirrusSearch

https://gerrit.wikimedia.org/r/381791

Change 381791 merged by jenkins-bot:
[integration/config@master] Run mediawiki-core-qunit-selenium-jessie for CirrusSearch

https://gerrit.wikimedia.org/r/381791

I have set up mediawiki-core-qunit-selenium-jessie job to run webdriverio patches in experimental pipeline.

Example:

https://gerrit.wikimedia.org/r/#/c/381785/
https://integration.wikimedia.org/ci/job/mediawiki-core-qunit-selenium-jessie/7662/console

@EBernhardson told me on IRC that you are already working on this, but instead of mocha using cucumber. I am not sure how much mediawiki-core-qunit-selenium-jessie would be useful for that currently, but we should be able to make it work. Let me know.

Change 381785 abandoned by Zfilipin:
WIP The first Selenium test in Node.js

Reason:
just a sample

https://gerrit.wikimedia.org/r/381785

@EBernhardson is there a reason CirrusSearch uses mwbot instead of nodemw (like core and a few extensions)?

@Jdrewniak setup the framework for running tests in nodejs, not sure why in particular mwbot was chosen but jan can probably comment. After using it a bit i can say that it's batch() method in particular is quite convenient as our setup creates lots of different pages with varied content to be searched for.

@zeljkofilipin There are a few reasons I chose mwbot instead of nodemw:

  • Promises. mwbot does not support promises out of the box, and Cucumber.js, the testing framework used in CirrusSearch, is heavily promise-based, so mwbot worked with it much better than nodemw
  • Batch edit/update convenience functions. Like Erik mentioned, there's a lot of article setup with CirrusSearch, (generating many test articles) so having batch edit functionality was a big plus
  • Generally nicer logging with mwbot (goes well with the batching)

I spent a little time today with nodemw and i concur with @Jdrewniak. For promises we can probably get away with something like bluebird.promisifyAll, but the batching functionality is key. The combination of succinct call structure, auto-magic concurrency, and operation specific error handling all built into mwbot's batch functionality is incredibly useful for our hooks that set everything up. Error handling for nodemw specifically is pretty difficult, as it doesn't actually return enough information. For example if i delete a page that doesn't exist i get back an error object containing the string Error returned by API: The page you specified doesn't exist.. This is a localized string that varies when we test against, for example, our language specific wikis. I really need the original response which has {code: 'missingtitle', info: 'The page you specified doesn\'t exist', ...}

I think it would be good to pick one tool and standardize on it. I do not have any experience with mwbot. I will play try it out (T181284).

debt added a comment.Nov 27 2017, 4:49 PM

Having a standardized tool is a very good thing! :) @zeljkofilipin let us know if using mwbot works for you.

debt closed this task as Resolved.Dec 15 2017, 6:31 PM
debt claimed this task.

Out of curiosity, this task is marked as resolved, but I see that Ruby Selenium tests are not deleted?

https://phabricator.wikimedia.org/diffusion/ECIR/browse/master/tests/browser/

Change 399178 had a related patch set uploaded (by DCausse; owner: DCausse):
[mediawiki/extensions/CirrusSearch@master] Remove ruby browser tests

https://gerrit.wikimedia.org/r/399178

Change 399184 had a related patch set uploaded (by Zfilipin; owner: Zfilipin):
[integration/config@master] Do not run Ruby Jenkins jobs for CirrusSearch

https://gerrit.wikimedia.org/r/399184

Change 399184 merged by jenkins-bot:
[integration/config@master] Do not run Ruby Jenkins jobs for CirrusSearch

https://gerrit.wikimedia.org/r/399184

Change 399178 merged by jenkins-bot:
[mediawiki/extensions/CirrusSearch@master] Remove Ruby browser tests

https://gerrit.wikimedia.org/r/399178

Change 381785 restored by Zfilipin:
WIP The first Selenium test in Node.js

https://gerrit.wikimedia.org/r/381785