Page MenuHomePhabricator

Outreachy 2020 - 2021 Proposal: Evaluate Microsoft Playwright as replacement for our browser automation
Closed, ResolvedPublic



Name: Harriet Ayugi
Time zone: UTC +3
Location: Uganda
Working hours: 2:00 pm to 11:00 pm UTC +3


Currently Wikimedia supports a large number of repositories as well as extensions. In order to ensure good code practices across all these repos, an extensive amount of testing is performed.

Currently, WebdriverIO is used as the browser automation framework for implementing end to end tests across multiple repos.

However with the recent increase in awareness about end to end testing, a number of equally competitive non-selenium solutions have been introduced. Two such include Puppeteer and Cypress. As a part of this years Google Summer of Code program, an extensive evaluation of Puppeteer was performed. Unfortunately owing to Cypress not being able to run properly on CI. Evaluation of Cypress had to be dropped.

One of the contenders being considered at the time was Microsoft Playwright. Seeing the recent developments in the framework and the rapid surge in popularity. We would like to evaluate it against WebdriverIO as a potential replacement for our browser automation framework.

Possible Implementation Methods

The following implementation methods are proposed;

  • Implement existing tests across moderately sized repo's in Microsoft Playwright
Why Is This Necessary?

As for the current implementation, WebdriverIO and Selenium are being used. However, the Wikimedia team is looking for a modern end to end testing tool. Microsoft Playwright is one of the most modern tool with the ability to allow cross browser automation testing, moreover with a recent stable versions release.

How Will The Proposed Features Benefit Wikimedia Projects?

Given that Microsoft Playwright is a modern testing automation tool with many new added testing features that the other testing tools may not have, this new features will enable faster testing, cross browser testing among other benefits.

Do You See Any Risks/Concerns Involved In implementing The Planned Features?

Microsoft Playwright is a very new automation testing tool with it's first public version stable realise at version v0.10.0. What this means is that it's more of an experimental tool for the community and the API is prone to breaking which might greatly affect the evaluation process. There could also be compatibility-related issues.


@Soham @zeljkofilipin @Vidhi-Mody

Dec 1 to Dec 09Community bonding, Request access to repositories and development tools on Phabricator, get familarised with the Code bases, plan, research and setup the Microsoft Playwright testing framework.
Dec 10 to Jan 10Start implementation of existing tests across moderately sized repo's in Microsoft Playwright
Jan 11 to Jan 20Introduce video recording and screenshots of tests
Jan 21 to Feb 5Benchmark Playwright against Puppeteer (using previous implementation) and WebdriverIO.
Feb 6 to Feb 20Re-run a short evaluation on Cypress (Depending upon whether the issues with Cypress are resolved)
Feb 21 to 2 MarchFeedback, Clear technical debts and Write reports
  • Blog posts on my progress every week.
  • Blog posts on my experience with Microsoft Playwright and FOSS-related topics at least once every two weeks.
  • Regular communication with my mentors and other community members.

I will select the available repo to work on, make changes and create patches to be merged periodically as tasks get done.
I will be online on Zulip in my working hours ( 2:00 pm to 11:00 pm UTC +3) to collaborate with the mentors.
I will use Phabricator for managing bugs and subtasks.

About Me

I'm a software developer with experience in building applications using JavaScript, NodeJs, Reactjs, Testing framework like Cypress and Jest.

Event Timeline

DannyS712 added subscribers: Aklapper, DannyS712.

@Aklapper is the bugwrangler (see and should probably be left on the task unless he chooses to unsubscribe

@Harriet I have noticed you are yet to complete the application on the Outreachy website. Please do so before the deadline tomorrow. The phabricator task is just a part of the proposal. Until you submit the proposal via Outreachy, we wont be able to consider your application

Noted @Soham, I'm working on it. Thanks