=Synopsis
Mediawiki is an important software component of Wikimedia that powers a large number of companies and organisations, including Wikipedia. Because of the various organisations and websites that depend upon it, it is necessary to evaluate software components under various expected and unexpected conditions so as to ensure delivery of optimal quality code.
To facilitate frequent and repetitive regression and E2E testing of a web application, a robust test approach would involve automating checks at the browser level. This reduces the manual testing effort and allows for detection of defects at an early stage. Wikimedia does this using a browser automation framework called [[ https://webdriver.io | WebdriverIO ]].
**//In this task, through experimentation, we compare one of the two most popular browser automation frameworks, focusing on primarily two points(speed and stability)//**
- **WebdriverIO:** WebdriverIO has been around for quite some time now, and has established itself as one of the popular web browser automation tool. It supports a wide range of browsers (Chrome, Firefox, Safari …), and uses Javascript. It uses Selenium under the hood.
- **Cypress:** Cypress is much newer browser automation framework (early 2015). It was primarily developed for the purpose of automating and simplifying front-end tests and development and solve most frequently encountered problems by QA and testing teams. The remarkable claim that Cypress makes is to be able to test anything that runs in a browser. It supports Javascript (Node.js) and does not use Selenium under the hood.
=Why look for an alternative?
A quick glance through various automation frameworks in trend at present times:
{F31733682}
source: https://www.npmtrends.com/cypress-vs-puppeteer-vs-webdriverio-vs-nightwatch
Though highly reliable, WebdriverIO comes with its own set of cons. The major one being:
- Difficulty in setup. A brief glance through Mediawiki's "package.json" file highlights the dependency tree for WebdriverIO
```
"chromedriver": "73.0.0"
"karma": "3.1.4"
"karma-chrome-launcher": "2.2.0"
"karma-firefox-launcher": "1.1.0"
"@wdio/cli": "5.13.2"
"@wdio/devtools-service": "5.13.2"
"@wdio/dot-reporter": "5.13.2"
"@wdio/junit-reporter": "5.13.2"
"@wdio/local-runner": "5.13.2"
"@wdio/mocha-framework": "5.13.2"
"@wdio/sauce-service": "5.13.2"
"@wdio/sync": "5.13.2"
"wdio-chromedriver-service": "5.0.2"
"wdio-mediawiki": "file:tests/selenium/wdio-mediawiki"
"webdriverio": "5.13.2
```
- Flaky tests: Flaky tests lead to false negatives which can confuse the team and increase the time and effort needed in debugging.
- No consistent device emulation features
- Difficulty upgrading: Since WebdriverIO does not update in sync with Selenium, it makes it very difficult to upgrade WebDriverIO from one major version to another without breaking the existing code.
- Scattered documentation
=Setup
Assuming Node.js is installed, setting up Cypress is pretty straight forward: just run `npm install --save-dev cypress`. This will install Cypress and all it's dependencies.
=Test Implementation
> While migrating test cases from WebdriverIO to Cypress, the primary goal I had in mind was to implement this migration while making as little structural changes to existing code as possible.
> Though this part needed more efforts than just simply copy pasting code from documentation, it was worth it!! The end result was a code migrated to a much richer browser automation framework without sacrificing the existing code structure. This is a plus for developers who have previously worked on WebdriverIO with Mediawiki to still be able to understand the code without knowing Cypress.
>
**Below sample code tests for a user to be able to create account and then login with the newly created account**
#### WebdriverIO
`tests/selenium/wdio-mediawiki/Page.js`
```
const querystring = require( 'querystring' );
/**
* Based on http://webdriver.io/guide/testrunner/pageobjects.html
*/
class Page {
openTitle( title, query = {}, fragment = '' ) {
query.title = title;
browser.url(
browser.config.baseUrl + '/index.php?' +
querystring.stringify( query ) +
( fragment ? ( '#' + fragment ) : '' )
);
}
}
module.exports = Page;
```
`tests/selenium/pageobjects/createaccount.page.js`
```
const Page = require( 'wdio-mediawiki/Page' );
class CreateAccountPage extends Page {
get username() { return $( '#wpName2' ); }
get password() { return $( '#wpPassword2' ); }
get confirmPassword() { return $( '#wpRetype' ); }
get create() { return $( '#wpCreateaccount' ); }
get heading() { return $( '.firstHeading' ); }
open() {
super.openTitle( 'Special:CreateAccount' );
}
createAccount( username, password ) {
this.open();
this.username.setValue( username );
this.password.setValue( password );
this.confirmPassword.setValue( password );
this.create.click();
}
}
module.exports = new CreateAccountPage();
```
`tests/selenium/specs/user.js`
```
javascript
const assert = require( 'assert' );
const CreateAccountPage = require( '../pageobjects/createaccount.page' );
const Util = require( 'wdio-mediawiki/Util' );
describe( 'User', function () {
let password, username
beforeEach( function () {
browser.deleteAllCookies();
username = Util.getTestString( 'User-' );
password = Util.getTestString();
} );
it( 'should be able to create account', function () {
// create
CreateAccountPage.createAccount( username, password );
// check
assert.strictEqual( CreateAccountPage.heading.getText(), `Welcome, ${username}!` );
} );
} );
```
#### Cypress
`tests/cypress/cypress-mediawiki/Page.js`
```
const querystring = require( 'querystring' );
class Page {
/**
* Navigate the browser to a given page.
*/
openTitle( title, query = {}, fragment = '' ) {
query.title = title;
cy.visit( 'index.php?' +
querystring.stringify( query ) +
( fragment ? ( '#' + fragment ) : '' )
);
}
}
module.exports = Page;
```
`tests/cypress/pageobjects/createaccount.page.js`
```
const Page = require( '../cypress-mediawiki/Page' );
class CreateAccountPage extends Page {
constructor() {
super();
this.username = '#wpName2';
this.password = '#wpPassword2';
this.confirmPassword = '#wpRetype';
this.create = '#wpCreateaccount';
this.heading = '#firstHeading';
}
open() {
this.openTitle( 'Special:CreateAccount' );
}
createAccount( username, password ) {
this.open();
cy.get( this.username ).type( username );
cy.get( this.password ).type( password );
cy.get( this.confirmPassword ).type( password );
cy.get( this.create ).click();
}
}
module.exports = new CreateAccountPage();
```
`tests/cypress/integration/login.spec.js`
```
const Api = require( '../cypress-mediawiki/Api' );
const Util = require( '../cypress-mediawiki/Util' );
const CreateAccountPage = require( '../pageobjects/createaccount.page' );
const UserLoginPage = require( '../cypress-mediawiki/LoginPage' );
describe( 'User', function () {
let username, password, bot;
before( async () => {
bot = await Api.bot();
} );
beforeEach( () => {
cy.clearCookies();
username = Util.getTestString( 'User-' );
password = Util.getTestString();
} );
it( 'should be able to create account', function () {
// create
CreateAccountPage.createAccount( username, password );
// check
cy.get( CreateAccountPage.heading ).contains( `Welcome, ${username}!` );
} );
it( 'should be able to log in @daily', function () {
// create
new Cypress.Promise( async () => {
await Api.createAccount( bot, username, password );
} );
// log in
UserLoginPage.login( username, password );
// check
cy.get( UserLoginPage.userPage ).contains( username );
} );
} );
```
=Conclusion
#### Why Cypress?
There are a tonne of features that Cypress comes included with that make it desirable over WebdriverIO:
- Super easy setup. Just a
```
npm i --save-dev cypress
```
- Time Travel: Cypress takes snapshots as your tests run. Hover over commands in the [[ https://docs.cypress.io/guides/core-concepts/test-runner.html#Command-Log | Command Log ]] to see exactly what happened at each step.
- Debuggability: Stop guessing why your tests are failing. [[ https://docs.cypress.io/guides/guides/debugging.html | Debug directly ]] from familiar tools like Developer Tools. Our readable errors and stack traces make debugging lightning fast.
- Automatic Waiting: Never add waits or sleeps to your tests. Cypress [[ https://docs.cypress.io/guides/core-concepts/introduction-to-cypress.html#Cypress-is-Not-Like-jQuery | automatically waits ]] for commands and assertions before moving on. No more async hell.
- Spies, Stubs, and Clocks: Verify and [[ https://docs.cypress.io/guides/guides/stubs-spies-and-clocks.html | control the behavior ]] of functions, server responses, or timers. The same functionality you love from unit testing is right at your fingertips.
- Network Traffic Control: Easily [[ https://docs.cypress.io/guides/guides/network-requests.html | control, stub, and test edge cases ]] without involving your server. You can stub network traffic however you like.
- Consistent Results: Our architecture doesn’t use Selenium or WebDriver. Say hello to fast, consistent and reliable tests that are flake-free.
- Screenshots and Videos: View screenshots taken automatically on failure, or videos of your entire test suite when run from the CLI.
- [[ https://docs.cypress.io/ | Rich Documentation ]]
- Growing community
- Superior [[ https://docs.cypress.io/api/commands/viewport.html#Syntax | device emulation capabilities
]]
**Experiences:**
- Setting up Cypress was super simple as compared to setting up WebdriverIO on my local machine
- The automatic awaiting for a page element was extremely helpful as there was no need to add manual time-intervals for letting the element load. The time for automatic awaits is highly configurable too!
- The included Dashboard for running in-browser tests was super user-friendly and it's integration with Chrome's developer tools made it super easy to fabricate test cases
- I tried running Cypress headless using the `cypress run` command and it ran super fast. This would be extremely important for dockerizing Cypress tests.
- The community on Github was super supportive in helping me out with setting up a custom configuration.
//To summarize, it will be extremely useful to evaluate Cypress as a possible replacement for WebdriverIO as a browser automation framework and discuss the scope of migrating it to eventually all of Wikimedia's front-end repos//