- The data below was collected by running Puppeteer and WebdriverIO tests for mediawiki/core and extensions/EntitySchema.
- Each repository has 15 test runs spanning over 2 separate machines totalling to a 30 tests overall.
- For measuring reliability, % tests failures for each run were computed and averaged over 30 test runs.
- Additionally, a short PoC to test for behaviour of the frameworks when running tests in parallel was also evaluated. This was done for an additional of 10 tests with 2 workers. Totalling to 20 tests per framework.
- Make a github gist containing all the metrics.