this seems likely to be an issue with SauceLabs and/or ChromeDriver, the @browser.window.resize_to command seems to have just stopped working for Chrome on SauceLabs, even though it works locally.
Confirmed as upstream bug: https://code.google.com/p/chromedriver/issues/detail?id=1036
| Project | Branch | Lines +/- | Subject | |
|---|---|---|---|---|
| mediawiki/extensions/MobileFrontend | master | +1 -1 | QA: Enable test on Chrome |
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Invalid | None | T108946 [Epic] Improve the development infrastructure | |||
| Resolved | zeljkofilipin | T94150 Fix scenarios that fail at en.wikipedia.beta.wmflabs.org or do not run them daily | |||
| Duplicate | None | T105985 Run Wikidata browser tests on testwikidata via Jenkins | |||
| Resolved | • Jonas | T88541 [DO NOT USE] Wikidata Browsertests (tracking) [superseded by #Browser-Tests] | |||
| Resolved | Tobi_WMDE_SW | T101497 [Story] Run browsertests regularly on test.wikidata.org via Jenkins | |||
| Resolved | JanZerebecki | T101498 [Task] Create a user with necessary privileges for browsertests on test.wikidata.org | |||
| Resolved | Tobi_WMDE_SW | T101499 [Task] Setup a Jenkins job to run Wikidata browsertests on test.wikidata.org | |||
| Resolved | Tobi_WMDE_SW | T92619 make Wikidata browsertests non-flaky (tracking) | |||
| Declined | Jdlrobson | T88288 Upstream: Issue with Chrome driver with resizing window |
@Jdlrobson: I do not know. You are the one that reported the bug, I did not have the time to investigate.
@chrismcmahon was working on this - he said that this was the reason MobileFrontend's browser tests for overlays were failing. I can't see any clues this is still happening so I assume this was an upstream bug and got fixed in new Chrome driver release
Given https://integration.wikimedia.org/ci/job/browsertests-MobileFrontend-SmokeTests-linux-chrome-sauce/153/testReport/(root)/Toggling%20sections/Opening_a_section_on_mobile/ and https://integration.wikimedia.org/ci/view/Mobile/job/browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce/682/testReport/junit/(root)/Toggling%20sections/Opening_a_section_on_mobile/ is failing I guess this is still an issue.
cc @dduvall
This appears to be a known issue with chromedriver and Chrome versions >= 40.[1] I was able to reproduce the bad behavior in SauceLabs with version 40,[2] and verify that the resize works as expected in version 39.[3]
One possible workaround would be to use browser_factory.override (MW-Selenium >= 1.3) to temporarily force the Chrome version to 39 for specific scenarios.[4] You'd probably want to implement it as a tag + hook to ensure that the configuration takes effect before the browser is opened—with MW-Selenium 1.x there's a factory system that delays the start of the browser session until it's first accessed with Environment#browser so this kind of runtime configuration is a lot easier than it used to be.[5]
Something like:
Before("@chrome-39") do
browser_factory(:chrome).override(version: '39')
end[1] https://code.google.com/p/chromedriver/issues/detail?id=1036
[2] SauceLabs, Chrome 40, https://saucelabs.com/tests/b63ee7a53d9b496f97e5199f1ad2c983
[3] SauceLabs, Chrome 39, https://saucelabs.com/tests/edfe4df30a6e45db98888d4775f8df94
[4] https://doc.wikimedia.org/rubygems/mediawiki-selenium/MediawikiSelenium/BrowserFactory/Base.html#override-instance_method
[5] https://doc.wikimedia.org/rubygems/mediawiki-selenium/MediawikiSelenium/Environment.html#browser-instance_method
We will discuss this task at the next browser tests triage meeting: https://www.mediawiki.org/wiki/Quality_Assurance/Browser_testing/Meetings/Notes
I'd rather wait for the fix in Chrome, but it would be good to link to the upstream bug in the subject of this task so we know when we can remove the @skip tag. Since it works in Firefox I think we have sufficient coverage.
In future when we upgrade versions of the drivers could we be more careful to communicate this and check they do not impact builds, as I remember when this bug initially came up we were very confused.
It wasn't an upgrade to any of the underlying libraries that we use that caused the issue but some interaction between chromedriver and the latest version of Chrome.
I also seem to see the same issue with firefox on saucelabs. Causes like 1 or 2 failures per run for wikidata.
I can only assume one year later that given this hasn't come up again this is no longer a problem.
AFAIK it happens all the time on saucelabs, but as a non-free service I can't fix saucelabs.