We decided to switch from SauceLabs + Nightwatch to Cypress in (for now) GitHub Actions for Browser testing. (Included in the discussions and approved by @conny-kawohl_WMDE)
The main motivations for this step are:
- Cypress supports native mocking of requests and we already have a [nice PoC that showcases that request mocking](https://github.com/wmde/query-builder/pull/177)
- Nightwatch would need plugins or a mock-server
- We have to move the repository to Gerrit/WMF-Gittiles at some point (T272130#6792307) and it is unclear if it is even possible to keep using SauceLabs with Gerrit
- at the very least the effort to integrate Cypress into the WMF CI stack will be smaller, there even exists some prior work (T230729)
- Cypress supports running with multiple browser width and allows for visual regression testing at multiple break-points
- We are not using this feature yet, but it might be a nice to have in the future
- Cypress can run in Headless mode, which might make it more suitable for CI
- This project is a good candidate to test Cypress under real conditions. It should be suitable to give us data about test-flakiness in day-today use.
- that data can then inform the decision on whether we should migrate more projects to Cypress
The main drawback:
- Cypress only supports testing on Chrome and Firefox, not Internet Explorer 11 and Safari like SauceLabs/Nightwatch does
- however, that is in line with our current testing standards for projects hosted on Gerrit/Gittiles which are only tested on Chrome anyway
- QueryBuilder does not support Internet Explorer 11 at all
- QB uses browserlist in Babel and PostCSS to ensure compatibility with all required Browsers, so Safari //should// be fine as well 馃