Page MenuHomePhabricator

Measure reliability and speed of various browser drivers
Open, MediumPublic

Description

Status

  1. The data below was collected by running Puppeteer and WebdriverIO tests for mediawiki/core and extensions/EntitySchema.
  2. Each repository has 15 test runs spanning over 2 separate machines totalling to a 30 tests overall.
  3. For measuring reliability, % tests failures for each run were computed and averaged over 30 test runs.

mediawiki/core

toolreliabilitytime (s)
Puppeteer100%214
WebdriverIO100%316
WebdriverIO Async100%322

extensions/EntitySchema

toolreliabilitytime (s)
Puppeteer100%342
WebdriverIO100%517
  1. Additionally, a short PoC to test for behaviour of the frameworks when running tests in parallel was also evaluated. This was done for an additional of 10 tests with 2 workers. Totalling to 20 tests per framework.

    mediawiki/core
toolreliabilitytime (s)
Puppeteer100%193
WebdriverIO100%297
WebdriverIO Async82%311

TODO

  • Make a github gist containing all the metrics.

Links

Event Timeline

Restricted Application added a subscriber: Aklapper. ยท View Herald TranscriptJul 31 2020, 11:05 AM
Soham updated the task description. (Show Details)