Albert Sison (Sauce Labs Help Center)
Feb 17, 3:22 PM PST
Thanks for the update and please let us know if re-running the failed tests work for you. In case you're wondering, I have attached screenshots of "Internal Server Error" and "The Sauce VMs failed to start the browser or device." error rate for jobs rans by all users for the past 30 days.
Mon, Feb 20
Thu, Feb 16
As far as I can see, all you need is already implemented. Let me know if I have missed something.
338146 is an example patch. Test it with:
After some testing, looks like it is already supported.
Update to Selenium 3 fixes tests on Ruby 2.4.
Upgrade to Selenium 3 was easy: https://gerrit.wikimedia.org/r/#/c/336824/
Research is done. From what I can see, I prefer webdriverio over webdriverjs and nightwatch.
After some research, webdriverio looks like the way to go.
Sauce labs support ticket (not public): https://support.saucelabs.com/hc/en-us/requests/35513
Wed, Feb 15
Unfortunately, looks like running sequentially is supported only for executing strategy classic, but we use yaml :(
The trouble with recent data is that we had a lot of trouble with unrelated MediawikiApi::ApiError problems. Now that that is resolved, I have created a new job selenium-Wikibase-336632-4 that will run every 4 hours (because it needs over 3 hours for a run). I will let it run for a day or two.
I did a bit of investigation, thinking that builds targeting test wiki fail more for some reason, but that does not seem to be the case. Builds failing with MediawikiApi::ApiError are a couple of problems unrelated to this one, and should be ignored.
Tue, Feb 14
MediawikiApi::ApiError is unrelated problem (T157665) and should be ignored, so only last 5 runs are relevant. In the last 5 runs, both Linux and Windows did not have any problems. Linux runs are faster than Windows, Mac is the slowest. Mac runs also failed more often.
Number of failures in the last 10 runs
The setup is finished. The next step is to actually write tests.
Jenkins job is failing:
Oops (again), looks like IP is blocked, not the user. :)
I think the user is blocked, not the IP.
Oops, looks like it is not related:
The state is that I did not have the time to work on this yet :(
Fri, Feb 10
Selenium user can create accounts via the API at beta cluster without captcha:
Thu, Feb 9
This broke a lot of browser tests, not just CentralAuth. :(
Reproduced on my machine when targeting beta:
Can not reproduce on my machine targeting vagrant VM.
Wed, Feb 8
MEDIAWIKI_CAPTCHA_BYPASS_PASSWORD is set up in job-templates-selenium.yaml.
Build #268 is the last one that is green:
Test jobs are created and running:
Date: Wed, 8 Feb 2017 15:55:55 +0100
This Jenkins job has 211 tests and runs once a day. Each test starts a
Sauce Labs job. I have investigated jobs 253-258, so 6 days, or 1266 tests.
"The Sauce VMs failed to start the browser or device" error has appeared 5
times. 5/1266 = 0.0039 or 0.4%.
I will run the same job on Windows and Mac and see if there are any
Albert Sison (Sauce Labs Help Center)
Feb 3, 1:20 PM PST
Thanks for writing in to Sauce Labs support. Regarding this specific ”Internal Server Error", this means the VM to run the test started and then become unresponsive. The VM appears to have hit a resource limit and crashed. It’s hard to tell exactly what limit was hit. Later our services tried to restart the VM, but by that point the test was no longer registered as an active job (i.e. it had “gone stale”), and the job failed. This is a type of failure we’ve seen before: where a VM hits a resource limit and crashes. It’s a known problem where we don’t know the exact cause, not enough to prevent it from happening again. In the engineer’s words, it is a “rare but expected scenario”. More information about this error can be found here: https://wiki.saucelabs.com/display/DOCS/Common+Error+Messages#CommonErrorMessages-InternalServerError
As for the "The connection with your VM was lost and your job can't complete." error, if you only get this message rarely and randomly, it is probably a fluke on our end caused by an infrastructure blip as mentioned on our error page: https://wiki.saucelabs.com/display/DOCS/Common+Error+Messages#CommonErrorMessages-TheConnectionwithYourVirtualMachinewasLostandYourJobCan'tComplete
For "The Sauce VMs failed to start the browser or device." error, what percentage of your tests are affected by this specific error? I noticed all the Sauce job URLs provided were testing against Linux/Chrome 48. For diagnostic purposes, can you try testing against a different OS such as Windows 10 or OSX and let us know if you notice any improvements? Thanks for your patience and understanding.
Fri, Feb 3
I have reported the problem to Sauce Labs support.
@hashar Good idea about checking the limit, but we never reach it.
Thu, Feb 2
Sauce Labs status says everything is fine on their end, there were no incidents in the last week. I took a quick look, the rest of selenium jobs are very rarely failing with Selenium::WebDriver::Error::UnknownError.