Status
- The data below was collected by running Puppeteer and WebdriverIO tests for mediawiki/core and extensions/EntitySchema.
- Each repository has 15 test runs spanning over 2 separate machines totalling to a 30 tests overall.
- For measuring reliability, % tests failures for each run were computed and averaged over 30 test runs.
mediawiki/core
| tool | reliability | time (s) |
| Puppeteer | 100% | 214 |
| WebdriverIO | 100% | 316 |
| WebdriverIO Async | 100% | 322 |
extensions/EntitySchema
| tool | reliability | time (s) |
| Puppeteer | 100% | 342 |
| WebdriverIO | 100% | 517 |
- Additionally, a short PoC to test for behaviour of the frameworks when running tests in parallel was also evaluated. This was done for an additional of 10 tests with 2 workers. Totalling to 20 tests per framework.
mediawiki/core
| tool | reliability | time (s) |
| Puppeteer | 100% | 193 |
| WebdriverIO | 100% | 297 |
| WebdriverIO Async | 82% | 311 |
TODO
- Make a github gist containing all the metrics.