Page MenuHomePhabricator

Measure reliability and speed of various browser drivers
Closed, DeclinedPublic

Description

Status

  1. The data below was collected by running Puppeteer and WebdriverIO tests for mediawiki/core and extensions/EntitySchema.
  2. Each repository has 15 test runs spanning over 2 separate machines totalling to a 30 tests overall.
  3. For measuring reliability, % tests failures for each run were computed and averaged over 30 test runs.

mediawiki/core

toolreliabilitytime (s)
Puppeteer100%214
WebdriverIO100%316
WebdriverIO Async100%322

extensions/EntitySchema

toolreliabilitytime (s)
Puppeteer100%342
WebdriverIO100%517
  1. Additionally, a short PoC to test for behaviour of the frameworks when running tests in parallel was also evaluated. This was done for an additional of 10 tests with 2 workers. Totalling to 20 tests per framework.

    mediawiki/core
toolreliabilitytime (s)
Puppeteer100%193
WebdriverIO100%297
WebdriverIO Async82%311

TODO

  • Make a github gist containing all the metrics.

Links

Event Timeline

Soham updated the task description. (Show Details)
zeljkofilipin raised the priority of this task from Medium to Needs Triage.May 17 2021, 1:52 PM