Page MenuHomePhabricator

GSoC 2020 Proposal: Evaluate WebdriverIO replacements for our browser automation framework
Closed, ResolvedPublic

Description

Profile Information

Name: Soham Parekh
IRC nickname on Freenode: alqaholic007
Email: mail@sohamp.dev / soham.parekh1998@gmail.com
Webpage: https://sohamp.dev
Resume:


GitHub: https://github.com/AlQaholic007
Institute: University of Mumbai
Location: Mumbai, India
Typical working hours: 4 PM to 1 AM (UTC+5:30). Will be available anytime on prior notice.
Timezone: UTC+5:30 (IST)

Synopsis

Motivation
https://filipin.eu/selenium-javascript
https://www.cypress.io/how-it-works/
https://blog.scottlogic.com/2020/01/13/selenium-vs-puppeteer.html

What question are we chasing?
Should we and if so how much work would it take to migrate all the automation tests to an alternative automation framework other than WebdriverIO

Overview
The MediaWiki software is used by thousands of websites and companies, including Wikipedia.
Currently all the frontend automation tests for MediaWiki are written in Javascript using WebDriverIO which uses Selenium under the hood. While WebDriverIO offers many advantages like cross-browser support, large and continuously growing community, etc, some major hurdles while using WebDriverIO such as:

  • non-trivial set-up
  • flaky tests
  • poor documentation
  • painful debugging and syntax which is painful to grasp.

Additionally WebDriverIO does not synchronize with the Selenium updates so upgrading it to a newer version is a painstaking task so much so that it has an entire GSoC project related to it for the 2020 edition, namely T247844: Upgrade WebdriverIO to the latest version for all repositories, indicating that it is indeed a non-trivial and time-consuming task, not to mention all the familiar bugs and quirks that come with Selenium.

It should therefore be in our interest to evaluate other front-end automation frameworks as replacements for WebDriverIO and the approximate the time taken to span this migration over all repos. This will not only help us not worry about the overheads with manually testing the upgraded version of an automated testing tool but also better maintain the projects while staying up to date with the top of the edge automation frameworks. Moreover this can be migrated to other front-end repos within Wikimedia as well after evaluations are evidenced based on developer experience, stability of tests, etc.

Migrate!! How do we do that?

What are the alternatives then?

Here is a brief comparision of monthly downloads of these frameworks over npm which showcase growing popularity of Cypress and Puppeteer as test automation frameworks


source: https://www.npmtrends.com/cypress-vs-webdriverio-vs-puppeteer

Here is a brief comparision of some of the pros and cons of these frameworks(not exhaustive and may differ with time)

WebdriverIOCypressPuppeteer
Pros- support for most BDD and TDD test frameworks- rich documentation- easy setup
- enthusiastic developer community- fast(<20ms response time)- tremendous flexibility
- ‘webdrivercss’ to compare css stylings of an element in the webpage- built on top of Mocha and Chai- backed and supported by team behind Chromium
- Works with any testing framework(Jasmine,Mocha,Cucumber) or assertion library(Chai for Mocha)- ability to stub responses- written in Typescript
- support for Appium- active community on Github- Promise based
- Auto Reload acts as real user- fast
- commands automatically retry assertions- can run parallely
- super easy setup- support for device emulation
Cons- poor documentation- no parallel yet- poor documentation since its spread over multiple places
- too many dependencies to set up- no support for switching tabs- Firefox support still in beta
- tests are flaky- relatively new- smaller community
- no full upload support- promise-based syntax seems unnatural and hard to grasp

source: several

There are a tonne of alternatives out there but based up on reading several articles and reviews, Cypress.io and Puppeteer turn out as the most natural choices. While Cypress.io is increasingly growing in terms of popularity because of not using Selenium under the hood, Puppeteer is backed and constantly maintained by the team that maintains Chromium and is extremely flexible. Both however are highly flexible but fall out on cross-browser support. However they check all the boxes in our requirements to support Chromium-powered browsers and Firefox. This project aims to evaluate Cypress and Puppeteer as alternative test-automation frameworks based upon various factors such as:

  • stability of tests
  • ease of installation, developer experience
  • community support
  • speed of test runs
  • configuration options

and if positive, estimate the work needed to migrate all repositories that currently use WebdriverIO to the chosen framework.

The various tasks currently to be worked upon are:

Mentor: @zeljkofilipin @Jpita
Have you contacted your mentor already? Yes I have already connected to them.

Timeline

PeriodTask
May 4 to May 31Community Bonding Period
June 1 to June 7Familiarize with the code-base and the existing tests and test high level functionalities of MediaWiki in Cypress
June 8 to June 14Establish different configs and script commands and implement tests that run against different environments like local, test and production.
June 15 to June 21Implement the remaining end to end tests in Cypress to test MediaWikis overall functionalities and test Cypress's device emulation capabilities on MobileFrontend Extension
June 22 to June 28Explore integrating the dockerized Cypress tests with existing Jenkins pipelines.
June 29 to July 3Phase 1 evaluations: Draft a comprehensive overview of Cypress vs WebdriverIO particular to MediaWiki with the help of mentors. This will conclude the subtask T230729: Evaluate Cypress as a WebdriverIO replacement for our browser automation framework
July 4 to July 10Set up puppeteer with jest and jest-puppeteer and write preliminary tests for high level functionalities with Puppeteer.
July 11 to July 17Implement end to end tests in Puppeteer to test MedaWiki’s overall functionalities and test Puppeteer's device emulation capabilities with MobileFrontend Extension
July 18 to July 26Dockerize the tests and explore integration of Puppeteer tests with existing Jenkins pipelines.
July 27 to July 31Phase 2 Evaluations: Draft a comprehensive overview of Puppeteer vs WebdriverIO particular to MediaWiki with the help of mentors. This concludes the sub-task T247843: Evaluate Puppeteer as a WebdriverIO replacement for our browser automation framework
August 1 to August 7By now we should have eliminated one of the two alternatives based on developer experience, stability of tests, etc. Set up ability to run tests in specific environments using environment variable overrides.
August 8 to August 15Completely integrate the tests of the chosen framework with the current Jenkins pipeline in dockerized fashion with support for env variable overrides for running tests in different environments and document the tasks so far.
August 16 to August 23Ensure if all the tests work with the right configurations correctly. Discuss the outcome of the evaluations and the scope to migrate the automation framework to other front-end repos if the results are positive
August 24 to August 31Final commit. Try to evaluate the outcome of the project(how much work for migrating to an alternative framework?) with mentors and document the evaluations made on both frameworks.
September 1 to September 7Mentors submit final evaluations
September 8Final results of Google Summer of Code 2020 announced
Future WorkContinue to work on bugs and propose to move evaluations to other front-end repos by consulting mentors. T248779: Set up CI for MediaWiki-Docker also has driven my interest. Mentor students in GCI and GSoC.

Note: This is only a tentative timeline that I've come up with since this project is sort of a Proof of Concept(PoC) based project and may change as per the recommendations given by the mentors.

Milestones

Phase I evaluation
Revolves around the sub-task: T230729: Evaluate Cypress as a WebdriverIO replacement for our browser automation framework

  • Familiarity with MediaWiki's code-base and testing workflows
  • Understand existing test suites and implementation of new test cases
  • Exploring with configurations, commands and plugins in Cypress.
  • Implementation of MediaWiki's E2E tests in Cypress
  • Testing device emulation capabilities of Cypress with MobileFrontend Extension
  • Explore the option of configuring test suites(local, test or production env) with environment variable overrides
  • Integrate the test suites with existing Jenkins pipeline
  • Evaluation of performance metrics with the help of mentors
  • Documentation of migrations and evaluations with the help of mentors

Phase II evaluation
Revolves around the sub-task: T247843: Evaluate Puppeteer as a WebdriverIO replacement for our browser automation framework

  • Setting up Puppeteer with jest and jest-puppeteer.
  • Exploring various configuration options in Puppeteer
  • Implementation of MediaWiki's E2E tests in Puppeteer
  • Testing device emulation capabilities of Cypress with MobileFrontend Extension
  • Explore the option of configuring test suites(local, test or production env) with environment variable overrides
  • Integrate the test suites with existing Jenkins pipeline
  • Evaluation of performance metrics with the help of mentors
  • Documentation of migrations and evaluations with the help of mentors

Final evaluation

  • Collating results from the evaluation will help us eliminate one of the two frameworks.
  • Document the evaluations by using facts and conclusions.
  • Estimate the time for migrating to chosen framework over all front-end repos and discuss it's scope with mentors.
  • Submit a final report based on the overall documentation of evaluations

Participation

  • Will be working over 40 hours a week
  • Available on IRC and Zulip during my working hours.
  • Available on Email outside working hours.
  • Update the status of the issues being worked upon as a comment in Phabricator.
  • Maintain and publish a bi-weekly blog that helps summarize our experiences with the frameworks along with their pros and cons based on personal experiences.

About Me

I am a pre-final year undergraduate student, pursuing B.E. in Computer Engineering from University of Mumbai, Mumbai, India. I have been contributing to open source projects for quite some time now. Currently, I am a maintainer for the npm package github-api. I have developed a deep and keen interest in working on open source projects since they expose me to work on real world problems, get acquainted with several coding practices used by different organisations and evaluate my coding skills in production.

I first heard about GSoC from a senior who had participated in GSoC 2019 under Cern HSF. He motivated me to spend my year to plan to work on a project under an organisation I like. I have been using Wikipedia, Wikibooks and Wikiquote since my school days and when I saw Wikimedia Foundation enlisted a participating organisation in this year's GSoC, I was ecstatic. Being able to contribute to a software that backs these tools instills in me immense joy and feeling of wanting to give back to this community I’ve grown up with. I am a big advocate of the mission of the Wikimedia Foundation as on a personal level, coming from a developing nation like India, I am well-versed with disadvantages that the underprivileged face due to the costs and logistics involved with gaining quality knowledge.

My reason for choosing this project in particular has to do with me struggling to choose the right set of automation tools for implementing E2E tests on my own final year project. The exposure that I gain to the in-trend front-end automation frameworks will help me make better decisions with my own projects as well and with these skills, I plan to continue contributing to Wikimedia.

Past Experience

I have worked on several different projects that span a wide range of technologies like Javacript, Node.js, Docker, Kubernetes, Ansible, Jenkins, ELK Stack, PHP, Python, NodeJS, C++, WebGL, Java, React, Django, etc. I am also well-versed with several database technologies like MySQL, PostgreSQL, SQLite, MariaDB and MongoDB.
I heavily rely on git as a version control for most of my projects as well as other open source projects so much so that I also ended up working on building "UnBelievaBot" , a highly configurable Github Bot written in NodeJS that integrates with Telegram, Discord and Slack.

Microtasks Completed:

All my other open source contributions and personal projects are available on https://github.com/AlQaholic007

Event Timeline

Soham created this task.Mar 28 2020, 11:27 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 28 2020, 11:27 AM
Soham updated the task description. (Show Details)Mar 28 2020, 12:00 PM
Soham updated the task description. (Show Details)Mar 28 2020, 12:06 PM
Soham updated the task description. (Show Details)Mar 28 2020, 3:11 PM
Soham updated the task description. (Show Details)
Soham updated the task description. (Show Details)Mar 28 2020, 3:19 PM
Soham updated the task description. (Show Details)
Soham updated the task description. (Show Details)Mar 28 2020, 3:29 PM
Soham updated the task description. (Show Details)Mar 28 2020, 3:33 PM
Soham updated the task description. (Show Details)Mar 30 2020, 5:18 AM
Soham updated the task description. (Show Details)Mar 30 2020, 5:21 AM
Soham updated the task description. (Show Details)Mar 30 2020, 5:23 AM
Soham updated the task description. (Show Details)
Soham updated the task description. (Show Details)Mar 30 2020, 5:36 AM
Soham updated the task description. (Show Details)Mar 30 2020, 11:13 AM
Soham updated the task description. (Show Details)Mar 30 2020, 11:41 AM
Soham updated the task description. (Show Details)Mar 30 2020, 12:04 PM
Soham updated the task description. (Show Details)Mar 30 2020, 12:34 PM
Soham updated the task description. (Show Details)Mar 30 2020, 1:37 PM
Soham updated the task description. (Show Details)Mar 30 2020, 1:40 PM
Soham updated the task description. (Show Details)Mar 30 2020, 1:45 PM
Soham updated the task description. (Show Details)Mar 30 2020, 1:48 PM
Soham updated the task description. (Show Details)Mar 30 2020, 1:50 PM
Soham updated the task description. (Show Details)Mar 30 2020, 2:50 PM
Soham updated the task description. (Show Details)Mar 30 2020, 5:31 PM
Soham updated the task description. (Show Details)
Soham updated the task description. (Show Details)Mar 30 2020, 5:40 PM
Soham updated the task description. (Show Details)
Soham updated the task description. (Show Details)Mar 30 2020, 5:53 PM
Soham updated the task description. (Show Details)Apr 14 2020, 5:24 AM
Soham updated the task description. (Show Details)
zeljkofilipin moved this task from Project ⚙️ to Mentor 🎓 on the User-zeljkofilipin board.
zeljkofilipin moved this task from Backlog to Project on the User-Soham board.

@Pavithraes @srishakatux Can we resolve this task? Does it have to be open during the internship? We're using T247835 to keep track of the project.

zeljkofilipin closed this task as Resolved.Sep 9 2020, 8:22 AM

@Soham please reopen if there's anything left to do here.

@Soham Thanks for your work and congratulations on a successful GSoC!
@Jpita, @zeljkofilipin Thank you for mentoring ^_^