move wikidata browsertests to not use saucelabs
Closed, ResolvedPublic

Description

It seems saucelabs is broken: T88288 . Try to make the wikidata jobs not fail always by moving to use a local (on our jenkins slave) browser instance instead.

JanZerebecki updated the task description. (Show Details)
JanZerebecki raised the priority of this task from to Needs Triage.
JanZerebecki added a subscriber: JanZerebecki.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 21 2015, 1:41 PM

This should be really easy to do. Our framework supports running tests on a local browser or Sauce Labs. Let me know if you need help with this, or if you would like to pair on it.

zeljkofilipin triaged this task as Normal priority.Oct 21 2015, 4:55 PM
zeljkofilipin added a project: Easy.

Change 247901 had a related patch set uploaded (by JanZerebecki):
[WIP] Run Wikidata browsertests without saucelabs

https://gerrit.wikimedia.org/r/247901

It seems not setting --format Cucumber::Formatter::Sauce as in the patch does not cause it to not use saucelabs. How do I instruct the framework to use a local browser instead?

@JanZerebecki
AFAIK it uses Saucelabs whenever SAUCE_ONDEMAND_USERNAME and SAUCE_ONDEMAND_ACCESS_KEY are set. Unsetting these should make it use the local browser.

Yes that seems to disable saucelabs. But now I get "unable to obtain stable firefox connection in 60 seconds" as a failure for the second scenario:
http://raita.wmflabs.org/#projects/https%3A%2F%2Fgithub.com%2Fwmde%2FWikidataBrowserTests.git/builds/FLBroL2mRIOCJAGEQ3QFvw
https://gerrit.wikimedia.org/r/#/c/247901/2/

I think the only thing left is to tell the machine you want to run the browser headless.

export HEADLESS=true

https://github.com/wikimedia/mediawiki-selenium#headless-mode

Yes that seems to work.

After looking at the results running without saucelabs:
I filed: T117418: Ensure ChromeDriver is installed for jobs that run Selenium tests
The jobs run in about half the time.
It seems the runs have more failures than with saucelabs.

Jonas added a comment.Nov 2 2015, 3:08 PM

Is this the right link to see the results?
So it's mostly timeouts?

This sadly made me laugh:

Scenario: Edit sitelink
Unexpected modal dialog (text: A script on this page may be busy, or it may have stopped responding. You can stop the script now, open the script in the debugger, or let the script continue.

A timeout will be also be reported when a link can not be found when the instruction was wait for that link to appear. So it is not that simple. Most of them seem not to be "A script on this page may be busy, or it may have stopped responding.".

For now lets run the jobs both with and without saucelabs until we figure out how to make one of them not flaky.

That's the nice thing about saucelabs: you can have a look at the test-run in a video to figure out what went wrong.

If you can figure that out, then what does go wrong with the last few failures on sauce labs?

@JanZerebecki I think that's blocked on T115500

raita.wmflabs.org shows the correct link.

Change 247901 merged by jenkins-bot:
Additionally run Wikidata browsertests without saucelabs

https://gerrit.wikimedia.org/r/247901

JanZerebecki closed this task as Resolved.Nov 3 2015, 6:31 PM
JanZerebecki claimed this task.

Now the old jobs are switched back to saucelabs again and new jobs are created that do not use saucelabs. There are many different problems at the same time, so it is unclear if saucelabs bug is even a problem. Once we manage to get one of the jobs non-flaky, we'll revisit if we run them on saucelabs.