We want to stop maintaining two separate stacks for browser tests.
- we will disable ruby tests that are making the daily tests run against beta fail.
- we start rewriting ruby browser tests one by one (in sub-tasks), keeping in mind that some of these tests might be not needed and can be discarded completely. That's to be figured out on each case separately while working on them.
[relevant info on current ruby tests]
This job used to be green in the early 2018, now seems to be failing again since couple of weeks, e.g. https://integration.wikimedia.org/ci/view/Selenium/job/selenium-Wikibase-chrome/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=DebianJessie%20&&%20contintLabsSlave/lastBuild/testReport/
There seems to be several types of failures of individual tests:
- timed out after 15 seconds, Element was not visible in 15 seconds (Watir::Wait::TimeoutError) -> to check if it is a hiccups, or indicating some DOM change leading to not finding an element?
- You do not have the permissions needed to carry out this action. (permissiondenied) (MediawikiApi::ApiError) -> is it some edit ratio limit maybe?
- possibly more types of errors?
First step would probably to identify all "classes" of failures that happen there, and then dig into this individually. I expect there are no 80+ different failures, but just a handful.