It seems saucelabs is broken: T88288 . Try to make the wikidata jobs not fail always by moving to use a local (on our jenkins slave) browser instance instead.
Description
Description
Details
Details
| Subject | Repo | Branch | Lines +/- | |
|---|---|---|---|---|
| Additionally run Wikidata browsertests without saucelabs | integration/config | master | +37 -2 |
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Invalid | None | T108946 [Epic] Improve the development infrastructure | |||
| Resolved | zeljkofilipin | T94150 Fix scenarios that fail at en.wikipedia.beta.wmflabs.org or do not run them daily | |||
| Duplicate | None | T105985 Run Wikidata browser tests on testwikidata via Jenkins | |||
| Resolved | • Jonas | T88541 [DO NOT USE] Wikidata Browsertests (tracking) [superseded by #Browser-Tests] | |||
| Resolved | Tobi_WMDE_SW | T101497 [Story] Run browsertests regularly on test.wikidata.org via Jenkins | |||
| Resolved | JanZerebecki | T101498 [Task] Create a user with necessary privileges for browsertests on test.wikidata.org | |||
| Resolved | Tobi_WMDE_SW | T101499 [Task] Setup a Jenkins job to run Wikidata browsertests on test.wikidata.org | |||
| Resolved | Tobi_WMDE_SW | T92619 make Wikidata browsertests non-flaky (tracking) | |||
| Resolved | JanZerebecki | T116166 move wikidata browsertests to not use saucelabs |
Event Timeline
Comment Actions
This should be really easy to do. Our framework supports running tests on a local browser or Sauce Labs. Let me know if you need help with this, or if you would like to pair on it.
Comment Actions
Change 247901 had a related patch set uploaded (by JanZerebecki):
[WIP] Run Wikidata browsertests without saucelabs
Comment Actions
It seems not setting --format Cucumber::Formatter::Sauce as in the patch does not cause it to not use saucelabs. How do I instruct the framework to use a local browser instead?
Comment Actions
@JanZerebecki
AFAIK it uses Saucelabs whenever SAUCE_ONDEMAND_USERNAME and SAUCE_ONDEMAND_ACCESS_KEY are set. Unsetting these should make it use the local browser.
Comment Actions
Yes that seems to disable saucelabs. But now I get "unable to obtain stable firefox connection in 60 seconds" as a failure for the second scenario:
http://raita.wmflabs.org/#projects/https%3A%2F%2Fgithub.com%2Fwmde%2FWikidataBrowserTests.git/builds/FLBroL2mRIOCJAGEQ3QFvw
https://gerrit.wikimedia.org/r/#/c/247901/2/
Comment Actions
I think the only thing left is to tell the machine you want to run the browser headless.
export HEADLESS=true
https://github.com/wikimedia/mediawiki-selenium#headless-mode
Comment Actions
After looking at the results running without saucelabs:
I filed: T117418: Ensure ChromeDriver is installed for jobs that run Selenium tests
The jobs run in about half the time.
It seems the runs have more failures than with saucelabs.
Comment Actions
Is this the right link to see the results?
So it's mostly timeouts?
This sadly made me laugh:
Scenario: Edit sitelink Unexpected modal dialog (text: A script on this page may be busy, or it may have stopped responding. You can stop the script now, open the script in the debugger, or let the script continue.
Comment Actions
A timeout will be also be reported when a link can not be found when the instruction was wait for that link to appear. So it is not that simple. Most of them seem not to be "A script on this page may be busy, or it may have stopped responding.".
Comment Actions
For now lets run the jobs both with and without saucelabs until we figure out how to make one of them not flaky.
Comment Actions
That's the nice thing about saucelabs: you can have a look at the test-run in a video to figure out what went wrong.
Comment Actions
If you can figure that out, then what does go wrong with the last few failures on sauce labs?
Comment Actions
Change 247901 merged by jenkins-bot:
Additionally run Wikidata browsertests without saucelabs
Comment Actions
Now the old jobs are switched back to saucelabs again and new jobs are created that do not use saucelabs. There are many different problems at the same time, so it is unclear if saucelabs bug is even a problem. Once we manage to get one of the jobs non-flaky, we'll revisit if we run them on saucelabs.