Page MenuHomePhabricator

browsertests under phantomjs have failures on Jenkins
Closed, DeclinedPublic

Description

I have setup up browsertests in Wikimedia Jenkins setup (bug 54388). The first run console using phantomjs browser dies miserably with some "Unable to pick a platform for the provided browser (RuntimeError)" errors.

Full output:
https://integration.wikimedia.org/ci/job/qa-browsertests-run/14/console

48 scenarios (25 failed, 23 skipped)


Version: wmf-deployment
Severity: normal

Details

Reference
bz54489

Event Timeline

bzimport raised the priority of this task from to Needs Triage.Nov 22 2014, 2:12 AM
bzimport set Reference to bz54489.
bzimport added a subscriber: Unknown Object (MLST).

Blocks bug 53697 "trigger browser tests from Gerrit" since it would be nice to have tests passing :-]

As far as I understand it, console says phantomjs is not installed. For example:

https://integration.wikimedia.org/ci/job/qa-browsertests-run/lastCompletedBuild/consoleFull

...
03:47:27 Unable to find phantomjs executable. (Selenium::WebDriver::Error::WebDriverError)
..

I forgot to install phantomjs during my crazy night.

Next issue:

unable to connect to phantomjs @ http://127.0.0.1:8910
after 20 seconds (Selenium::WebDriver::Error::WebDriverError

That is supposed to be the port of the phantomjs Selenium driver

vendor/ruby/1.9.1/gems/selenium-webdriver-2.35.0 \

... /lib/selenium/webdriver/phantomjs/service.rb

12: DEFAULT_PORT = 8910

The timeout is caused by {{bug|45868}}, that is, instances cant not access the public IP address. One has to rewrite outgoing packets using iptables rules as a workaround:

iptables -t nat -I OUTPUT --dest 208.80.153.219 -j DNAT --to-dest 10.4.1.133
iptables -t nat -I OUTPUT --dest 208.80.153.242 -j DNAT --to-dest 10.4.0.211
iptables -t nat -I OUTPUT --dest 208.80.153.243 -j DNAT --to-dest 10.4.0.51
iptables -t nat -I OUTPUT --dest 208.80.153.244 -j DNAT --to-dest 10.4.0.48
iptables -t nat -I OUTPUT --dest 208.80.153.243 -j DNAT --to-dest 10.4.1.82

No more timeout: https://integration.wikimedia.org/ci/job/qa-browsertests-run/20/console

We know have some issues such as:

timed out after 5 seconds, waiting for {:text=>"Yes", :tag_name=>"span"} to become present (Watir::Wait::TimeoutError)

Error Message => 'Can't find variable: mw'

Željko told me it is because the pages do not look the same under phantomJS, thus the tests would need to be fixed :-)

Following a pair session with Željko, the tests can be flagged with a tag which we can then exclude when running tests. For ULS, I have introduced the tag @specific-settings which let us skip any tests that are not going to pass on a fresh wiki installation: https://gerrit.wikimedia.org/r/#/c/97487/

https://gerrit.wikimedia.org/r/#/c/97487/ was for ULS. This bug is for qa/browsertest.

The idea was that whenever a change is proposed (or merged?) for qa/browsertests, we would have run the whole test suites using phantomJS on the WMF Jenkins.

Lot of tests being migrated to extensions I don't think it make sense anymore. We might want to repurpose qa/browsertest as smoke tests targeting production which would be quite useful for monitoring.

So either wont fix or resolve later on :-D

In that case, resolved wontfix. We can open new bugs for specific repositories if needed.