Profile Information
Name: Soham Parekh
IRC nickname on Freenode: alqaholic007
Email: mail@sohamp.dev / soham.parekh1998@gmail.com
Webpage: https://sohamp.dev
Resume:
GitHub: https://github.com/AlQaholic007
Institute: University of Mumbai
Location: Mumbai, India
Typical working hours: 4 PM to 1 AM (UTC+5:30). Will be available anytime on prior notice.
Timezone: UTC+5:30 (IST)
Synopsis
Motivation
https://filipin.eu/selenium-javascript
https://www.cypress.io/how-it-works/
https://blog.scottlogic.com/2020/01/13/selenium-vs-puppeteer.html
What question are we chasing?
Should we and if so how much work would it take to migrate all the automation tests to an alternative automation framework other than WebdriverIO
Overview
The MediaWiki software is used by thousands of websites and companies, including Wikipedia.
Currently all the frontend automation tests for MediaWiki are written in Javascript using WebDriverIO which uses Selenium under the hood. While WebDriverIO offers many advantages like cross-browser support, large and continuously growing community, etc, some major hurdles while using WebDriverIO such as:
- non-trivial set-up
- flaky tests
- poor documentation
- painful debugging and syntax which is painful to grasp.
Additionally WebDriverIO does not synchronize with the Selenium updates so upgrading it to a newer version is a painstaking task so much so that it has an entire GSoC project related to it for the 2020 edition, namely T247844: Upgrade WebdriverIO to v6 in all repositories, indicating that it is indeed a non-trivial and time-consuming task, not to mention all the familiar bugs and quirks that come with Selenium.
It should therefore be in our interest to evaluate other front-end automation frameworks as replacements for WebDriverIO and the approximate the time taken to span this migration over all repos. This will not only help us not worry about the overheads with manually testing the upgraded version of an automated testing tool but also better maintain the projects while staying up to date with the top of the edge automation frameworks. Moreover this can be migrated to other front-end repos within Wikimedia as well after evaluations are evidenced based on developer experience, stability of tests, etc.
Migrate!! How do we do that?
What are the alternatives then?
Here is a brief comparision of monthly downloads of these frameworks over npm which showcase growing popularity of Cypress and Puppeteer as test automation frameworks
source: https://www.npmtrends.com/cypress-vs-webdriverio-vs-puppeteer
Here is a brief comparision of some of the pros and cons of these frameworks(not exhaustive and may differ with time)
| WebdriverIO | Cypress | Puppeteer | |
| Pros | - support for most BDD and TDD test frameworks | - rich documentation | - easy setup |
| - enthusiastic developer community | - fast(<20ms response time) | - tremendous flexibility | |
| - ‘webdrivercss’ to compare css stylings of an element in the webpage | - built on top of Mocha and Chai | - backed and supported by team behind Chromium | |
| - Works with any testing framework(Jasmine,Mocha,Cucumber) or assertion library(Chai for Mocha) | - ability to stub responses | - written in Typescript | |
| - support for Appium | - active community on Github | - Promise based | |
| - Auto Reload acts as real user | - fast | ||
| - commands automatically retry assertions | - can run parallely | ||
| - super easy setup | - support for device emulation | ||
| Cons | - poor documentation | - no parallel yet | - poor documentation since its spread over multiple places |
| - too many dependencies to set up | - no support for switching tabs | - Firefox support still in beta | |
| - tests are flaky | - relatively new | - smaller community | |
| - no full upload support | - promise-based syntax seems unnatural and hard to grasp | ||
source: several
There are a tonne of alternatives out there but based up on reading several articles and reviews, Cypress.io and Puppeteer turn out as the most natural choices. While Cypress.io is increasingly growing in terms of popularity because of not using Selenium under the hood, Puppeteer is backed and constantly maintained by the team that maintains Chromium and is extremely flexible. Both however are highly flexible but fall out on cross-browser support. However they check all the boxes in our requirements to support Chromium-powered browsers and Firefox. This project aims to evaluate Cypress and Puppeteer as alternative test-automation frameworks based upon various factors such as:
- stability of tests
- ease of installation, developer experience
- community support
- speed of test runs
- configuration options
and if positive, estimate the work needed to migrate all repositories that currently use WebdriverIO to the chosen framework.
The various tasks currently to be worked upon are:
- T230729: Evaluate Cypress as a WebdriverIO replacement for our browser automation framework
- T247843: Evaluate Puppeteer as a WebdriverIO replacement for our browser automation framework
Mentor: @zeljkofilipin @Jpita
Have you contacted your mentor already? Yes I have already connected to them.
Timeline
| Period | Task |
|---|---|
| May 4 to May 31 | Community Bonding Period |
| June 1 to June 7 | Familiarize with the code-base and the existing tests and test high level functionalities of MediaWiki in Cypress |
| June 8 to June 14 | Establish different configs and script commands and implement tests that run against different environments like local, test and production. |
| June 15 to June 21 | Implement the remaining end to end tests in Cypress to test MediaWikis overall functionalities and test Cypress's device emulation capabilities on MobileFrontend Extension |
| June 22 to June 28 | Explore integrating the dockerized Cypress tests with existing Jenkins pipelines. |
| June 29 to July 3 | Phase 1 evaluations: Draft a comprehensive overview of Cypress vs WebdriverIO particular to MediaWiki with the help of mentors. This will conclude the subtask T230729: Evaluate Cypress as a WebdriverIO replacement for our browser automation framework |
| July 4 to July 10 | Set up puppeteer with jest and jest-puppeteer and write preliminary tests for high level functionalities with Puppeteer. |
| July 11 to July 17 | Implement end to end tests in Puppeteer to test MedaWiki’s overall functionalities and test Puppeteer's device emulation capabilities with MobileFrontend Extension |
| July 18 to July 26 | Dockerize the tests and explore integration of Puppeteer tests with existing Jenkins pipelines. |
| July 27 to July 31 | Phase 2 Evaluations: Draft a comprehensive overview of Puppeteer vs WebdriverIO particular to MediaWiki with the help of mentors. This concludes the sub-task T247843: Evaluate Puppeteer as a WebdriverIO replacement for our browser automation framework |
| August 1 to August 7 | By now we should have eliminated one of the two alternatives based on developer experience, stability of tests, etc. Set up ability to run tests in specific environments using environment variable overrides. |
| August 8 to August 15 | Completely integrate the tests of the chosen framework with the current Jenkins pipeline in dockerized fashion with support for env variable overrides for running tests in different environments and document the tasks so far. |
| August 16 to August 23 | Ensure if all the tests work with the right configurations correctly. Discuss the outcome of the evaluations and the scope to migrate the automation framework to other front-end repos if the results are positive |
| August 24 to August 31 | Final commit. Try to evaluate the outcome of the project(how much work for migrating to an alternative framework?) with mentors and document the evaluations made on both frameworks. |
| September 1 to September 7 | Mentors submit final evaluations |
| September 8 | Final results of Google Summer of Code 2020 announced |
| Future Work | Continue to work on bugs and propose to move evaluations to other front-end repos by consulting mentors. T248779: Set up CI for mwcli also has driven my interest. Mentor students in GCI and GSoC. |
Note: This is only a tentative timeline that I've come up with since this project is sort of a Proof of Concept(PoC) based project and may change as per the recommendations given by the mentors.
Milestones
Phase I evaluation
Revolves around the sub-task: T230729: Evaluate Cypress as a WebdriverIO replacement for our browser automation framework
- Familiarity with MediaWiki's code-base and testing workflows
- Understand existing test suites and implementation of new test cases
- Exploring with configurations, commands and plugins in Cypress.
- Implementation of MediaWiki's E2E tests in Cypress
- Testing device emulation capabilities of Cypress with MobileFrontend Extension
- Explore the option of configuring test suites(local, test or production env) with environment variable overrides
- Integrate the test suites with existing Jenkins pipeline
- Evaluation of performance metrics with the help of mentors
- Documentation of migrations and evaluations with the help of mentors
Phase II evaluation
Revolves around the sub-task: T247843: Evaluate Puppeteer as a WebdriverIO replacement for our browser automation framework
- Setting up Puppeteer with jest and jest-puppeteer.
- Exploring various configuration options in Puppeteer
- Implementation of MediaWiki's E2E tests in Puppeteer
- Testing device emulation capabilities of Cypress with MobileFrontend Extension
- Explore the option of configuring test suites(local, test or production env) with environment variable overrides
- Integrate the test suites with existing Jenkins pipeline
- Evaluation of performance metrics with the help of mentors
- Documentation of migrations and evaluations with the help of mentors
Final evaluation
- Collating results from the evaluation will help us eliminate one of the two frameworks.
- Document the evaluations by using facts and conclusions.
- Estimate the time for migrating to chosen framework over all front-end repos and discuss it's scope with mentors.
- Submit a final report based on the overall documentation of evaluations
Participation
- Will be working over 40 hours a week
- Available on IRC and Zulip during my working hours.
- Available on Email outside working hours.
- Update the status of the issues being worked upon as a comment in Phabricator.
- Maintain and publish a bi-weekly blog that helps summarize our experiences with the frameworks along with their pros and cons based on personal experiences.
About Me
I am a pre-final year undergraduate student, pursuing B.E. in Computer Engineering from University of Mumbai, Mumbai, India. I have been contributing to open source projects for quite some time now. Currently, I am a maintainer for the npm package github-api. I have developed a deep and keen interest in working on open source projects since they expose me to work on real world problems, get acquainted with several coding practices used by different organisations and evaluate my coding skills in production.
I first heard about GSoC from a senior who had participated in GSoC 2019 under Cern HSF. He motivated me to spend my year to plan to work on a project under an organisation I like. I have been using Wikipedia, Wikibooks and Wikiquote since my school days and when I saw Wikimedia Foundation enlisted a participating organisation in this year's GSoC, I was ecstatic. Being able to contribute to a software that backs these tools instills in me immense joy and feeling of wanting to give back to this community I’ve grown up with. I am a big advocate of the mission of the Wikimedia Foundation as on a personal level, coming from a developing nation like India, I am well-versed with disadvantages that the underprivileged face due to the costs and logistics involved with gaining quality knowledge.
My reason for choosing this project in particular has to do with me struggling to choose the right set of automation tools for implementing E2E tests on my own final year project. The exposure that I gain to the in-trend front-end automation frameworks will help me make better decisions with my own projects as well and with these skills, I plan to continue contributing to Wikimedia.
Past Experience
I have worked on several different projects that span a wide range of technologies like Javacript, Node.js, Docker, Kubernetes, Ansible, Jenkins, ELK Stack, PHP, Python, NodeJS, C++, WebGL, Java, React, Django, etc. I am also well-versed with several database technologies like MySQL, PostgreSQL, SQLite, MariaDB and MongoDB.
I heavily rely on git as a version control for most of my projects as well as other open source projects so much so that I also ended up working on building "UnBelievaBot" , a highly configurable Github Bot written in NodeJS that integrates with Telegram, Discord and Slack.
Microtasks Completed:
- T248232: Micro-task: Make a trivial change to tests/selenium/README.md file of mediawiki/core repository
- T248016: Micro-task: Run Selenium tests on your machine
- T249323: Micro-task: Amend existing commit in Gerrit
- T248220: Micro-task: Create a simple test using Cypress
- T248219: Micro-task: Create a simple test using Puppeteer
- T249443: Cypress as a WebDriverIO replacement of Mediawiki's browser automation
- T249445: Puppeteer as a browser automation framework replacement for WebdriverIO
All my other open source contributions and personal projects are available on https://github.com/AlQaholic007