Page MenuHomePhabricator

Firefox v47 breaks mediawiki_selenium
Closed, ResolvedPublic

Description

Firefox v47 got pushed to trusty and it causes the selenium web driver to fail completely. Best course of action for now: disable mwext-mw-selenium job.

https://github.com/seleniumhq/selenium/issues/2110


Firefox 47.0 has been pushed to trusty-updates last week. Its release notes mentions:

Selenium WebDriver may cause Firefox to crash on startup, use Marionette WebDriver instead.

Marionette requires an executable geckodriver which is apparently not packaged in Ubuntu It is a Rust based software available at https://github.com/mozilla/geckodriver/

We can:

  • get our repository to change their integration env in environments.yaml to explicit use browser: chromium.
  • revert back to Firefox 46 and pin it to prevent future updates

Once we get mediawiki_selenium / geckodriver (T137540) we will be able to upgrade.

Links:

http://www.theautomatedtester.co.uk/blog/2016/selenium-webdriver-and-firefox-47.html
https://developer.mozilla.org/en-US/docs/Mozilla/QA/Marionette/WebDriver

Details

Related Gerrit Patches:
operations/puppet : production(DO NOT SUBMIT) contint: pin firefox to 46 on Trusty
mediawiki/extensions/Wikibase : masterchange selenium to use chrome instead of firefox
integration/config : masterzuul: make mwext-mw-selenium non voting

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Restricted Application added subscribers: Zppix, Aklapper. · View Herald TranscriptJun 10 2016, 2:34 PM

Change 293732 had a related patch set uploaded (by Hashar):
zuul: make mwext-mw-selenium non voting

https://gerrit.wikimedia.org/r/293732

Change 293732 merged by jenkins-bot:
zuul: make mwext-mw-selenium non voting

https://gerrit.wikimedia.org/r/293732

Change 293739 had a related patch set uploaded (by JanZerebecki):
(DO NOT SUBMIT) contint: pin firefox to 46 on Trusty

https://gerrit.wikimedia.org/r/293739

Mentioned in SAL [2016-06-10T15:15:38Z] <jzerebecki> T137561 integration-puppetmaster:/var/lib/git/operations/puppet# git fetch https://gerrit.wikimedia.org/r/operations/puppet refs/changes/39/293739/1 && git cherry-pick FETCH_HEAD

Mentioned in SAL [2016-06-10T15:18:31Z] <jzerebecki> T137561 integration-saltmaster:~# salt -v '*trusty*' cmd.run "cd /root/ && wget 'https://ubuntu.wikimedia.org/ubuntu/pool/main/f/firefox/firefox_46.0.1%2bbuild1-0ubuntu0.14.04.3_amd64.deb'"

Mentioned in SAL [2016-06-10T15:19:54Z] <jzerebecki> T137561 integration-saltmaster:~# salt -v '*trusty*' cmd.run "cd /root/ && dpkg -i firefox_46.0.1+build1-0ubuntu0.14.04.3_amd64.deb"

Mentioned in SAL [2016-06-10T15:29:34Z] <jzerebecki> T137561 integration-puppetmaster:/var/lib/git/operations/puppet# git reset --hard 1e1ff12b13b73b5c5e2015a72f51561f10b305d0

Change 293739 abandoned by JanZerebecki:
(DO NOT SUBMIT) contint: pin firefox to 46 on Trusty

Reason:
Didn't help.

https://gerrit.wikimedia.org/r/293739

Change 293742 had a related patch set uploaded (by JanZerebecki):
change selenium to use chrome instead of firefox

https://gerrit.wikimedia.org/r/293742

Change 293742 merged by jenkins-bot:
change selenium to use chrome instead of firefox

https://gerrit.wikimedia.org/r/293742

+2 about all what @JanZerebecki did ! That is how we handled Chromium pinning to v49. Kudos Jan!

zeljkofilipin triaged this task as High priority.
hashar updated the task description. (Show Details)Jun 13 2016, 10:19 AM

Change 293739 restored by Hashar:
(DO NOT SUBMIT) contint: pin firefox to 46 on Trusty

https://gerrit.wikimedia.org/r/293739

I am replaying the apt pinning to Firefox 46.0 which Jan did on Friday. Ie cherry picking https://gerrit.wikimedia.org/r/#/c/293739/3 and manually installing the package.

Mentioned in SAL [2016-06-13T10:35:36Z] <hashar> T137561 salt -v '*trusty*' cmd.run "cd /root/ && dpkg -i firefox_46.0.1+build1-0ubuntu0.14.04.3_amd64.deb"

Slaves are back on Firefox 46.0 . Gave a recheck on MultimediaViewer change https://gerrit.wikimedia.org/r/#/c/293703/ and it pass again.

hashar changed the task status from Open to Stalled.Jun 13 2016, 10:46 AM

Stalled pending update of mediawiki_selenium to Marionette T137540

Would probably need the geckodriver to be packaged / shipped on slaves, in such a case we would want a sub task.

zeljkofilipin lowered the priority of this task from High to Medium.Jun 14 2016, 2:19 PM
hashar reopened this task as Open.Jun 16 2016, 10:10 AM

That is tracking task, also blocked on having the geckodriver packaged and shipped to CI nodes T137797 :)

Zeljko provided a basic selenium script with marionette enabled. https://github.com/zeljkofilipin/firefox-marionette-geckodriver-rb which I slightly tweaked to run:

require "selenium-webdriver"
driver = Selenium::WebDriver.for :firefox, marionette: true
driver.navigate.to "https://google.com"
driver.navigate.to "https://en.wikipedia.org"
driver.navigate.refresh
driver.navigate.back
driver.quit

I have used the geckodriver I have I packaged for Debian unstable (T137797), installed it on Jessie and the scenario passed all fine. Note the script above does NOT exercise any advanced feature of the Webdriver protocol.

Another possiblity we have is to stick to Firefox ESR which is currently Firefox 45 and is available on Jessie.

From https://www.mozilla.org/en-US/firefox/organizations/faq/

We would have to switch the jobs from Trusty to Jessie.

We do not support Firefox 45 in the "Modern" grade, but according to https://www.mediawiki.org/wiki/Compatibility#Browsers it is in "Unkown".

Firefox 47.0.1 has been released on June 28th https://www.mozilla.org/en-US/firefox/47.0.1/releasenotes/

Fixed: Selenium WebDriver may cause Firefox to crash at startup

It is not yet in Ubuntu Trusty though.

The Ubuntu Mozilla Security Team is preparing a package for Firefox 47.0.1. It has been proposed in their staging area https://launchpad.net/~ubuntu-mozilla-security/+archive/ubuntu/ppa/+packages

firefox (47.0.1+build1-0ubuntu0.14.04.1) trusty-security; urgency=medium

  * New upstream stable release (47.0.1build1)
    - see USN-2993-2

 -- Chris Coulson <chris.coulson ... canonical.com>  Tue, 28 Jun 2016 21:02:29 +0100

Placeholder for the Ubuntu Security Notice 2993-2 (empty/undisclosed right now).

Running bundle update selenium-webdriver in a repository will update version of selenium-webdriver in Gemfile.lock to 2.53.4. All tests should run fine with Firefox 47.0.1. I have just tested it with mediawiki/core.

From: Željko Filipin
To: qa@lists.wikimedia.org
Date: Fri, 8 Jul 2016 13:41:27 +0200
Subject: Firefox + Selenium = </3
Hi,
as some of you have noticed, there were problems with Firefox (version 47)
and Selenium recently.
Firefox Driver is a Firefox extension that is bundled with Selenium.
Selenium uses it to drive Firefox. Mozilla decided that from Firefox 48 all
extensions have to be signed and for some reason Firefox Driver is not
signed. Something went wrong in Firefox 47 and Firefox Driver could no
longer drive Firefox.
We have fixed the problem by locking Firefox to version 46 in our CI.
The problem is fixed in Firefox 47.0.1. As soon as it is released as a
Debian package, we can update CI from 46 to it. Or we could stick with 45
Extended Support Release (ESR). Why 45? Please continue reading.
Mozilla has created Marionette "the next generation of FirefoxDriver",
and it is "nearing completion". But it's status page does not say it is
almost done. It looks to me that it is far from done. One of the people
working on it says it is 85% done. The problem with Marionette is that
some tests could work just fine, and others will fail. It is hard for me to
say when will it be ready.
Looks like Chrome is winning this browser war so it would make sense to
use Chrome/Chromium in our CI.
What do you think?

  1. Firefox 45
  2. Firefox 47.0.1
  3. I need to test in Firefox 48+! Please make it work it Marionette. I do

not care if some things are broken.

  1. Firefox is so 2015. Let's switch to Chrome.
  2. A combination of the above options. (Please specify.)
  3. Something else. (Please specify.)

Željko

Another interesting read: https://groups.google.com/forum/#!topic/selenium-developers/e_y4SyhZHX4

To quote:

  • The Firefox web driver allow arbitrary code execution in privileged context, so it is not going to be approved/signed.
  • Firefox Developer Edition supports unsigned addons.
Peter added a subscriber: Peter.Jul 8 2016, 7:48 PM
Peter added a comment.Jul 8 2016, 8:10 PM

I think it makes sense to test on the browser where we have most visitors, however we should make sure that Firefox also works, so we can switch back. I think our tests always should work/run with the latest version of the browser.

I've been following the issue close since it broke my side project and I've tried to understand why and what. The Selenium project wants to move away from shipping the driver inside of Selenium, that's why Mozilla ships Geckodriver outside the project (following Chrome/IE/Opera). It was just "unlucky" in a way it broke with 47 but good that it made it clear that Geckodriver isn't ready yet.

When 48 goes live early August Firefoxdriver is dropped and we need to use Geckodriver. When I've tested it for my other project, it didn't work out (Geckodriver 0.9), so if we plan to stay with Firefox we should start testing it asap. One thing though: Geckodriver doesn't support stable Firefox yet :/

zeljkofilipin removed zeljkofilipin as the assignee of this task.Jul 22 2016, 1:20 PM
zeljkofilipin moved this task from In progress to Backlog on the Continuous-Integration-Config board.
zeljkofilipin moved this task from In Progress to Next on the Browser-Tests-Infrastructure board.
zeljkofilipin moved this task from In Progress 🔨 to Next 🔜 on the User-zeljkofilipin board.

Thanks @Peter, I have talked with @hashar and the current plan is to support:

  • Firefox 46 (current situation)
    • maybe downgrade to 45 (ESR)
    • or 47.0.1 (latest version that works with firefoxdriver)
  • Chrome/Chromium
  • Firefox 48+ (experimental)
Peter added a comment.Sep 1 2016, 7:32 AM

FYI: Geckodriver 0.10 and the coming 1.0 release only works with Selenium 3.0 (only in beta right now), so there's a dependency in upgrading.

I am not sure what needs to be done here to resolved the task. Firefox 47.0.1 is released and working fine with the latest release of selenium-webdriver ruby gem. If nobody complains in a few days, I will resolve this task and continue working on T137540: Update mediawiki_selenium to use Marionette.

From a quick check-in with zelko. Since v48 (release-notes):

Add-ons that have not been verified and signed by Mozilla will not load.

And people complained about it on Selenium: https://github.com/SeleniumHQ/selenium/issues/2559

zeljkofilipin closed this task as Resolved.Sep 28 2016, 12:39 PM

Firefox 47.0.1 works fine with our current infrastructure. See T137540: Update mediawiki_selenium to use Marionette for related tasks.

We have the slaves pinned at 46 for now. Should we switch to 47.0.1 ?

Both 46 and 47.0.1 are fine (but please notice that 47.0.0 will break everything). If it is not a lot of work, please upgrade. :)

Change 293739 abandoned by Hashar:
(DO NOT SUBMIT) contint: pin firefox to 46 on Trusty

Reason:
trusty is gone \o/

https://gerrit.wikimedia.org/r/293739