🏃♂️ selenium-Wikibase-chrome/MEDIAWIKI_ENVIRONMENT=beta run takes 35-65 minutes. 20-57% of tests are failing (42-121 from 212). 💣 Failures should be fixed.
🏃♂️ selenium-Wikibase-chrome/MEDIAWIKI_ENVIRONMENT=test needs 9.5 hours to run 🐌 with just a few failures (4-16 out of 200 tests). If the duration of the job is acceptable, failures need to be fixed. Also, Jenkins job template should be updated so other jobs can specify a shorter timeout, since only this job needs such long timeout. If the duration is not acceptable, tests should be made faster.
Now Wikibase daily browser tests run using SauceLabs. And they randomly fail for reasons not related to the code but related to SauceLabs API. See T152963: Increase in failures caused by Saucelabs for details.
As soon as randomly failing tests are worse than no tests at all, we (Wikidata team) would like to have another daily job running those tests but on Jenkins (without using SauceLabs). Hopefully they will be stable.
PS: If they are stable we will probably like to kill SauceLabs job, but not right now.
Important changes to current job:
- Removed SAUCE_ONDEMAND_ACCESS_KEY.
- Instead of running on BrowserTests slaves, the job now runs on DebianJessie && contintLabsSlave.
The change in the shell script to take screenshots and record videos:
export SKIP_TMPFS=1 export HEADLESS=true export HEADLESS_DISPLAY=$((70 + EXECUTOR_NUMBER % 20)) export HEADLESS_DESTROY_AT_EXIT=true export HEADLESS_CAPTURE_PATH="$WORKSPACE/log" HEADLESS=true SCREENSHOT_FAILURES=true SCREENSHOT_FAILURES_PATH="$WORKSPACE/log" bundle exec rake selenium