NOTE: We already use Karma to run QUnit tests in Chrome and Firefox on the Jenkins slaves that run Linux. This task is about running tests in other Grade A browsers, such as Internet Explorer on Windows or Safari on Mac.
Karma supports SauceLabs via an official plugin and is easy to set up.
https://github.com/karma-runner/karma-sauce-launcher
We already use this in OOjs to test with various IE and Safari versions (in addition to Chrome/Firefox).
However the way it is set up in OOjs is not scalable and was more an unfinished experiment than anything else. Let's set this up properly.
- Figure out which SauceLabs account(s) to use.
- What accounts do we have?
- What is their quota?
- Should be configure different accounts per Jenkins slave?
- Provided credentials via Jenkins as environment variables.
- For local npm-testing, require developer to register a free SauceLabs account and add put the token in their environment variables (e.g. set in bashrc). Document this on mediawiki.org.
- Determine how reliable SauceLabs is and from which pipelines to run this:
- test pipeline only (voting as part of existing qunit job, but skipped in gate)
- gate pipeline only and via "check sauce" pragma (voting but skipped in main test for speed, similar to how we do zend/hvvm).
- test and postmerge pipeline (voting as part of existing qunit job, skipped in gate, reported to Gerrit post-merge to catch of regressions).