Page MenuHomePhabricator

Evaluate Puppeteer as a WebdriverIO replacement for wikimedia's browser automation framework
Closed, DeclinedPublic



Testing is a very important phase in the development process of all WikiMedia's repositories. However, unit tests are not sufficient for all situations needed to be tested. End-to-End tests need to be done. We can't navigate through webpages with unit tests, Wikimedia uses a browser Automation framework(WebdriverIO) and selenium for its End-to-End tests.
WebdriverIO has been around for a very long while and has a large community behind it, however, there are some useful alternatives out there that can be used as a replacement framework while still giving a better performance than WebdriverIO. Some of such replacement framework is puppeteer, a high-level browser automation API built atop Chrome DevTools Protocols.

Why should puppeteer be considered as a replacement for WebdriverIO?

Puppeteer has quite a number of advantages over WebdriverIO and a few of them are highlighted in bullet points below.

  • Easier to setup
  • Fewer Dependencies
  • Easy to Upgrade to a newer version
  • Handles device emulation properly
  • Access to Chrome Dev tools
  • Simpler syntax
  • Simpler JavaScript Execution
  • Easier to run across multiple browsers(Chrome and Firefox)

The above highlights some of the advantages puppeteer has over WebdriverIO but doesn't give in-depth or technical details about how this highlighted points make puppeteer a better choice. Some of the above listed have been grouped or categorized into sections in which a more in-depth detail is discussed below.


WebdriverIO has been around for a long time, it has a lot of community support and engagement when compared to puppeteer but over the years, the number of active users of puppeteer has grown considerably way beyond that of WebdriverIO. According to npm trends, Puppeteer is currently used by 70.5k repositories on Github andWebdriverIO is used in 21.7k repositories on Github.

pptr vs webdriverio.png (663×991 px, 79 KB)

Setup and Installation

Puppeteer requires no extra setup or tedious installation procedures, it can be installed via just one command line code(npm install puppeteer) unlike WebdriverIO.
WebdriverIO uses selenium as its browser automation framework, currently, the following WebdriverIO dependencies are being used on the MediaWiki/core repository

  • "@wdio/cli": "5.13.2"
  • "@wdio/devtools-service": "5.13.2"
  • "@wdio/dot-reporter": "5.13.2"
  • "@wdio/junit-reporter": "5.13.2"
  • "@wdio/local-runner": "5.13.2"
  • "@wdio/mocha-framework": "5.13.2"
  • "@wdio/sauce-service": "5.13.2"
  • "@wdio/sync": "5.13.2"
  • "wdio-chromedriver-service": "5.0.2"
  • "wdio-mediawiki": "file:tests/selenium/wdio-mediawiki"
  • "webdriverio": "5.13.2"

Depending on the browser that we want to use for our test, we would also need to install the dependencies of that browser's driver and the drivers themselves separately. Chrome for examples uses the chromium driver and Firefox uses the gecko driver which means we would still install dependencies such as the ones highlighted below

  • "karma": "3.1.4"
  • "karma-chrome-launcher": "2.2.0"
  • "karma-firefox-launcher": "1.1.0"
  • "chromedriver": "73.0.0"

Installing the above would take quite some time and it's just too many dependencies to work with.

In puppeteer's case, just the following dependencies are required to have it up and running;

  • "puppeteer": "^2.1.1"
  • "jest-puppeteer": "^4.4.0"

As highlighted above, puppeteer requires just a testing framework and a single dependency to be installed, this makes it very easy to set up and use. Also, no additional browser drivers need to be installed.

Device Emulation

Puppeteer handles device emulation way better than WebdriverIO. The table below shows a comparison of both.

OS supported Devices Supported
PuppeteerIOS, Android, WindowsSupports 70+ individual devices
WebdriverIOIOS, AndroidGeneralizes emulations to OS and not individual devices

We would notice that Puppeteer supports 3 OS while Puppeteer supports 2. Also, WebdriverIO does not emulate devices individually but rather generalizes its emulations to a particular OS which isn't so great as each device has a separate User-Agent, even if they are of the same Operating System.

Emulating devices with Puppeteer also has a cleaner syntax than Emulating with WebdriverIO

Access to Chrome Dev tools

Puppeteer by default has full access and control over all of the chrome Devtools functionality which makes it very useful when writing tests, It's gives access to functionalities like Network throttling, code coverage, browser's console, etc. However, WebdriverIO needs to use Puppeteer under the hood to access chrome dev tools.

Simple code examples

Here, I would be showing code samples to perform a very basic test on MediaWiki/core. The basic test performed will be to test for the account creation feature of MediaWiki/core. This is used to show the difference between the syntax of WebdriverIO and Puppeteer.



const querystring = require( 'querystring' );

 * Based on
class Page {
	openTitle( title, query = {}, fragment = '' ) {
		query.title = title;
			browser.config.baseUrl + '/index.php?' +
			querystring.stringify( query ) +
			( fragment ? ( '#' + fragment ) : '' )

module.exports = Page;


const Page = require( 'wdio-mediawiki/Page' );

class CreateAccountPage extends Page {
	get username() { return $( '#wpName2' ); }
	get password() { return $( '#wpPassword2' ); }
	get confirmPassword() { return $( '#wpRetype' ); }
	get create() { return $( '#wpCreateaccount' ); }
	get heading() { return $( '.firstHeading' ); }

	open() {
		super.openTitle( 'Special:CreateAccount' );

	createAccount( username, password ) {;
		this.username.setValue( username );
		this.password.setValue( password );
		this.confirmPassword.setValue( password );;

module.exports = new CreateAccountPage();


const assert = require( 'assert' );
const CreateAccountPage = require( '../pageobjects/' );
const Util = require( 'wdio-mediawiki/Util' );

describe( 'User', function () {
	let password, username
        beforeEach( function () {
		username = Util.getTestString( 'User-' );
		password = Util.getTestString();
	} );
        it( 'should be able to create account', function () {
		// create
		CreateAccountPage.createAccount( username, password );

		// check
		assert.strictEqual( CreateAccountPage.heading.getText(), `Welcome, ${username}!` );
	} );
} );



class CreateAccountPage{
	constructor(page){ = page;
		this.url = `${process.env.MW_SERVER || ''}/index.php?title=Special:CreateAccount&returnto=Main+Page`

	async createAccount(username, password){
			await'#wpName2', username);
			await'#wpPassword2', password);
			await'#wpRetype', password);
			console.log("The error is ", err);

		return$$eval('#firstHeading', h => h[0].textContent);


module.exports = (browser) => new CreateAccountPage(browser);


let CreateAccountPage = require('../pageObjects/createAccount');
let reporterObj = require('../Utils/reporter');
let credentials = require('../Utils/credentials');

describe('User should be able to create an account', () => {
	let page, credential;
	beforeAll(async () => {
        page = await global.__BROWSER__.newPage()
        credential = credentials.generateCredential('User');
        CreateAccountPage = await CreateAccountPage(page);

    it("It should be able to create an account", async () => {
        await CreateAccountPage.createAccount(credential.username, credential.password);
        let headingText = await CreateAccountPage.getHeadingText();
        expect(headingText).toBe(`Welcome, ${credential.username}!`);


Puppeteer has a whole lot of useful advantages over WebdriverIO, has a clean syntax, easier to work with and comes with more functionality. It would, therefore, be very beneficial if WebdriverIO is replaced with Puppeteer as the automated browser framework for Wikimedia

Event Timeline

@Gbahdeyboh: This either looks like a duplicate of T247843. Or maybe this is a subtask of T248679. Context welcome how this is related to other tasks. :)

@Aklapper This Is a solution to T247843 and not it's duplicate.

@Gbahdeyboh: I don't understand. :) If this is a "solution" then this maybe could have been added to the existing task, to avoid fragmentation of conversations?

Task T247843 has Micro tasks. On completion, usually what I do is mention it in the comment under the same task.

I submitted one of the Micro tasks T248219 by creating a new task for the solution T248648, I wanted to do the same for the submission of T247843 which was why I created this task since it's a bit bulky.

Also, I think the Micro tasks under T247843 are meant to get us prepared for the main task(T247843) itself right? So the main task requires a separate solution?

@Gbahdeyboh: This either looks like a duplicate of T247843.

At this point, multiple students are working on the same tasks. Having a task per student would be helpful for me. I've made this task a subtask of T247843.