Event Timeline
I made some progress in analysing the problem. It seems to occur because the browser window on SauceLabs gets resized during the testing session. When I start the tests on SauceLabs from my local machine the resize always happens at the beginning of the test. See:
https://saucelabs.com/tests/ab00ca5e947b46479acdd2d7d95bbdac
When I run the same test on Jenkins and SauceLabs the resize happens in the middle of the test. See:
https://saucelabs.com/jobs/83f77171d0a143fba92a7b94f879ceeb
https://integration.wikimedia.org/ci/view/BrowserTests/view/Wikidata/job/browsertests-Wikidata-WikidataTests-linux-chrome-sauce-DEBUG/1/testReport/junit/%28root%29/Setting%20ranks%20of%20statements/Click_the_rank_selector/
Note that the above use the old mediawiki_selenium gem. I also did some changes and experimented with mw_selenium 1.1.0. But still there was a resize in the middle of the process:
https://saucelabs.com/jobs/66b5a466fd764b15bcf878ce9c57f328
https://integration.wikimedia.org/ci/view/BrowserTests/view/Wikidata/job/browsertests-Wikidata-WikidataTests-linux-chrome-sauce-DEBUG/4/testReport/junit/%28root%29/Setting%20snaktypes%20of%20statements/Change_the_snaktype__outline_example_____novalue___/
I contacted SauceLabs to discuss the issue a few weeks ago. Even a simple test, that is just calling a website triggers a resize.
This occurred with mediawiki selenium 1.1.0.
https://saucelabs.com/tests/21510b0e2bbc4edba5d6b504ddeddd6a (between 0:26 and 0:30)
Dylan from SauceLabs forwarded the issue to the images team to investigate on the 4th of May. Poked him again about it today.
currently chrome browser tests run well see
probably due to chrome issues with resizing see
Dylan from SauceLabs just wrote me a message:
"Our developers have deployed a fix to prevent this from happening; Can you check and let us know if it’s still a problem?"
So maybe they _really_ fixed it. Will run several test though.
The issues seem fixed for Chrome. Firefox still fails with some of our tests due to resizing in the middle of the process.
I wrote Dylan again stating the problem.
Lets wait....
Dylan answered again he reproduced the issue himself and they are looking for a solution.
"I’ve done some testing myself and it almost appears like Firefox doesn’t render or resize correctly for the first 30 seconds of tests, under Linux."
When looking at old screencasts on SauceLabs this is happening at least since mid March 2015. Sadly older screencasts are not available anymore. Does anyone remember screencast older than that with the described effect?
@WMDE-Fisch: I think sauce labs deletes screenshots and videos older than 30 days, so even if you had the link, it would not help a lot.
We plan to use the browser tests to test after the Tuesday deployments on testwikidata, but that wont work as long as the tests keep failing/ are flaky.
How about running them fully on Wikimedia CI infrastructure as long as they don't work reliably on SauceLabs?
But more than the last 20 runs of https://integration.wikimedia.org/ci/job/browsertests-Wikidata-WikidataTests-linux-chrome-sauce/ failed.
True but
fails because of
https://phabricator.wikimedia.org/T106327
and
fails because of
https://phabricator.wikimedia.org/T102930
:-)
As mentioned in
https://phabricator.wikimedia.org/T95291
Firefox still resizes randomly but it does not break our tests anymore.
There are three non-flaky failing browser tests currently. They all pass locally against local instance for me.
- Edit sitelinks.Remove multiple sitelinks features/sitelinks.feature:204
- Creating statements.Cancel statement (outline example : | press the ESC key in the claim entity selector input field |) features/statement.feature:43
- Edit sitelinks.Edit sitelink features/sitelinks.feature:154