make Wikidata browsertests non-flaky (tracking)
Closed, ResolvedPublic

Related Objects

StatusAssignedTask
Resolvedzeljkofilipin
DuplicateNone
ResolvedJonas
ResolvedTobi_WMDE_SW
ResolvedTobi_WMDE_SW
ResolvedWMDE-Fisch
Resolvedthiemowmde
ResolvedJanZerebecki
Resolveddaniel
ResolvedTobi_WMDE_SW
Resolvedzeljkofilipin
DuplicateNone
Resolveddduvall
InvalidJanZerebecki
DeclinedJdlrobson
DeclinedNone
ResolvedJanZerebecki
ResolvedTobi_WMDE_SW
ResolvedTobi_WMDE_SW
ResolvedTobi_WMDE_SW
ResolvedJonas
ResolvedTobi_WMDE_SW
ResolvedNone
ResolvedTobi_WMDE_SW
Resolvedadrianheine
Duplicatezeljkofilipin
ResolvedJonas
There are a very large number of changes, so older changes are hidden. Show Older Changes

Removed Jenkins since that is for Jenkins itself. You want Browser-Tests instead :)

Lydia_Pintscher triaged this task as Normal priority.Mar 16 2015, 9:31 AM
Lydia_Pintscher moved this task from incoming to ready to go on the Wikidata board.
zeljkofilipin renamed this task from investigate failing browsertests on jenkins to investigate failing Wikidata browsertests on jenkins.
zeljkofilipin added a subscriber: Liuxinyu970226.
hashar removed a subscriber: hashar.Apr 8 2015, 2:16 PM

I made some progress in analysing the problem. It seems to occur because the browser window on SauceLabs gets resized during the testing session. When I start the tests on SauceLabs from my local machine the resize always happens at the beginning of the test. See:

https://saucelabs.com/tests/ab00ca5e947b46479acdd2d7d95bbdac

When I run the same test on Jenkins and SauceLabs the resize happens in the middle of the test. See:

https://saucelabs.com/jobs/83f77171d0a143fba92a7b94f879ceeb
https://integration.wikimedia.org/ci/view/BrowserTests/view/Wikidata/job/browsertests-Wikidata-WikidataTests-linux-chrome-sauce-DEBUG/1/testReport/junit/%28root%29/Setting%20ranks%20of%20statements/Click_the_rank_selector/

Note that the above use the old mediawiki_selenium gem. I also did some changes and experimented with mw_selenium 1.1.0. But still there was a resize in the middle of the process:

https://saucelabs.com/jobs/66b5a466fd764b15bcf878ce9c57f328
https://integration.wikimedia.org/ci/view/BrowserTests/view/Wikidata/job/browsertests-Wikidata-WikidataTests-linux-chrome-sauce-DEBUG/4/testReport/junit/%28root%29/Setting%20snaktypes%20of%20statements/Change_the_snaktype__outline_example_____novalue___/

@dduvall: is mediawiki-selenium doing anything with resizing the browser?

I contacted SauceLabs to discuss the issue a few weeks ago. Even a simple test, that is just calling a website triggers a resize.
This occurred with mediawiki selenium 1.1.0.

https://saucelabs.com/tests/21510b0e2bbc4edba5d6b504ddeddd6a (between 0:26 and 0:30)

Dylan from SauceLabs forwarded the issue to the images team to investigate on the 4th of May. Poked him again about it today.

Dylan from SauceLabs just wrote me a message:

"Our developers have deployed a fix to prevent this from happening; Can you check and let us know if it’s still a problem?"

So maybe they _really_ fixed it. Will run several test though.

The issues seem fixed for Chrome. Firefox still fails with some of our tests due to resizing in the middle of the process.

I wrote Dylan again stating the problem.

Lets wait....

Dylan answered again he reproduced the issue himself and they are looking for a solution.

"I’ve done some testing myself and it almost appears like Firefox doesn’t render or resize correctly for the first 30 seconds of tests, under Linux."

When looking at old screencasts on SauceLabs this is happening at least since mid March 2015. Sadly older screencasts are not available anymore. Does anyone remember screencast older than that with the described effect?

@WMDE-Fisch: I think sauce labs deletes screenshots and videos older than 30 days, so even if you had the link, it would not help a lot.

hoo raised the priority of this task from Normal to High.Jul 16 2015, 2:32 AM
hoo added a subscriber: hoo.

We plan to use the browser tests to test after the Tuesday deployments on testwikidata, but that wont work as long as the tests keep failing/ are flaky.

How about running them fully on Wikimedia CI infrastructure as long as they don't work reliably on SauceLabs?

The issues seem fixed for Chrome. Firefox still fails with some of our tests due to resizing in the middle of the process.

But more than the last 20 runs of https://integration.wikimedia.org/ci/job/browsertests-Wikidata-WikidataTests-linux-chrome-sauce/ failed.

As mentioned in

https://phabricator.wikimedia.org/T95291

Firefox still resizes randomly but it does not break our tests anymore.

Tobi_WMDE_SW updated the task description. (Show Details)Jul 28 2015, 7:49 AM
Jonas renamed this task from investigate failing Wikidata browsertests on jenkins to [Task] investigate failing Wikidata browsertests on jenkins.Aug 13 2015, 3:32 PM
JanZerebecki renamed this task from [Task] investigate failing Wikidata browsertests on jenkins to [tracking] make Wikidata browsertests non-flaky.Nov 4 2015, 11:01 AM
JanZerebecki updated the task description. (Show Details)

There are three non-flaky failing browser tests currently. They all pass locally against local instance for me.

  • Edit sitelinks.Remove multiple sitelinks features/sitelinks.feature:204
  • Creating statements.Cancel statement (outline example : | press the ESC key in the claim entity selector input field |) features/statement.feature:43
  • Edit sitelinks.Edit sitelink features/sitelinks.feature:154
Danny_B renamed this task from [tracking] make Wikidata browsertests non-flaky to make Wikidata browsertests non-flaky (tracking).May 27 2016, 5:47 PM