Current status:

$ export BROWSER_LABEL=phantomjs
z@imac:~/project/wmf/browsertests(master)$ bundle exec cucumber --tags @en.wikipedia.beta.wmflabs.org

...

Failing Scenarios:
cucumber features/aftv5.feature:23 # Scenario: Click yes and leave feedback
cucumber features/file.feature:9 # Scenario: Logged-in user goes to file that does not exist
cucumber features/math_readonly.feature:5 # Scenario: Display simple math
cucumber features/page.feature:7 # Scenario: Go to page that does not exist
cucumber features/page.feature:12 # Scenario: Prepare to start a new page using the URL
cucumber features/page.feature:22 # Scenario: Start a new page using the URL
cucumber features/page.feature:36 # Scenario: Move existing page dialog
cucumber features/page.feature:48 # Scenario: Move existing page
cucumber features/page_edit.feature:16 # Scenario: Read link
cucumber features/page_edit.feature:25 # Scenario: View History link
cucumber features/page_triage_readonly.feature:5 # Scenario: Check that NewPagesFeed has correct controls for anonymous user
cucumber features/page_triage_readonly.feature:14 # Scenario: Check set filters selection
cucumber features/pdf_readonly.feature:10 # Scenario: Click on Download as PDF link
cucumber features/print_export_menu_readonly.feature:9 # Scenario: Check for Print/export section Expanded
cucumber features/print_export_menu_readonly.feature:14 # Scenario: Print/export section collapses after it is expanded
cucumber features/uls_cog_sidebar_anon_user.feature:58 # Scenario: More languages
cucumber features/uls_ime.feature:4 # Scenario: User can see input method keyboard icon
cucumber features/uls_ime.feature:9 # Scenario: User can see a dropdown menu when they click on the IME icon

55 scenarios (18 failed, 37 passed)
266 steps (18 failed, 58 skipped, 190 passed)
13m53.749s

$ export BROWSER_LABEL=firefox
z@imac:~/project/wmf/browsertests(master)$ bundle exec cucumber --tags @en.wikipedia.beta.wmflabs.org

...

Failing Scenarios:
cucumber features/aftv5.feature:23 # Scenario: Click yes and leave feedback
cucumber features/page_triage_readonly.feature:5 # Scenario: Check that NewPagesFeed has correct controls for anonymous user
cucumber features/page_triage_readonly.feature:14 # Scenario: Check set filters selection
cucumber features/pdf_readonly.feature:10 # Scenario: Click on Download as PDF link
cucumber features/print_export_menu_readonly.feature:14 # Scenario: Print/export section collapses after it is expanded
cucumber features/uls_cog_sidebar_anon_user.feature:58 # Scenario: More languages
cucumber features/uls_ime.feature:4 # Scenario: User can see input method keyboard icon
cucumber features/uls_ime.feature:9 # Scenario: User can see a dropdown menu when they click on the IME icon

55 scenarios (8 failed, 47 passed)
266 steps (8 failed, 11 skipped, 247 passed)
17m22.890s

For some reason, more tests fail when they are run in parallel.

$ export BROWSER_LABEL=phantomjs
$ bundle exec parallel_cucumber features/ --test-options '--tags @en.wikipedia.beta.wmflabs.org'
4 processes for 26 features, ~ 6 features per process

...

55 scenarios (25 failed, 4 skipped, 26 passed)
266 steps (25 failed, 92 skipped, 149 passed)

Took 347.455641 seconds
Tests Failed

Number of failed tests does not raise when using Firefox and running the tests in parallel.

$ export BROWSER_LABEL=firefox
$ bundle exec parallel_cucumber features/ --test-options '--tags @en.wikipedia.beta.wmflabs.org'
4 processes for 26 features, ~ 6 features per process

...

55 scenarios (7 failed, 48 passed)
266 steps (7 failed, 11 skipped, 248 passed)

Took 396.053233 seconds
Tests Failed

tplavcic wrote:

I tried to do a non scientific approach and run the tests in firefox and phantomjs and if I got the same result for the same test (be it the same fail or pass) then I would conclude that the test is o.k. in phantomjs ;)
I took this approach because I don't have enough experience and i'm not really sure in what condition are the tests.
I'm not sure if that's good enough (please comment), but here are my results (#number means i have log for this):

STATUS:
aftv5.feature - last test doesn't pass on ff & ph #3 (timeout) - not sure but I would remove @phantomjs-bug tag because I get the same test result as on ff?
create_account_readonly.feature - ok on both
file.feature - ok on both
guided_tour.feature - ok on both
login.feature - last test doesn't pass on ff & ph #2 (timeout)
login_sul.feature - ok on both
math_readonly - ff ok, ph error #19 (unable to locate element)
page_edit.feature - test2: ok on both, en.beta... ff one error #5 (modal dialog present), ph another #6 (timeout, element not visible)
page.feature - ff ok, ph error #20 (expected #, multiple: unable to locate element)
page_triage_readonly.feature - test2 ok on both, on en.beta server returns API page error on both #18 (timeout)
pdf_readonly.feature - ok
preferences_appearance_readonly.feature - ok
preferences_datetime_readonly.feature - ok
print_export_menu_readonly.feature - same error ff & ph #7 (expected visible)
search.feature - ok
uls_accept_language.feature - should be tagged @phantomjs-bug (runs only in ff and chrome)
uls_cog_sidebar_anon_user.feature - ok
uls_cog_sidebar_logged_user.feature - ok
uls.feature - same error ff & ph #9 (unable to locate element)
uls_ime.feature - same error ff & ph #8 (expected visible)
upload_wizard.feature - ff ok on test2, ph error #10 (timeout)
visual_editor.feature - same error ff & ph #11 (timeout)
visual_editor_links.feature - ff ok on en.beta - test2 error #14 (timeout), phantomjs errors #13 (Net::ReadTimeout) #14 (timeout)
visual_editor_references.feature - ff ok on en.beta - test2 error #15 (timeout), ph error #15 on both env
visual_editor_transclusion.feature - ff one error #16 (element not attached to DOM), ph another #17 (timeout)
wikilove.feature - ok

Tomislav,

I have removed keyword "easy" from this bug since I no longer think it is an easy bug, but feel free to continue working on it. :)

As you have correctly noticed, at the moment we have tests that are failing for certain browser/MEDIAWIKI_URL combinations.

The ideal scenario would be:

• a feature/scenario is passing for at least one browser (usually Firefox or Chrome)
• run the same feature/scenario using phantomjs
• investigate the failures (if any)

and then

• fix the failures for phantomjs (if possible)

or

• tag the feature/scenario with @phantomjs-bug

The prerequisite is to get a feature/scenario passing for Firefox/Chrome, meaning that failing tests should be investigates and fixed (if possible) or tagged @firefox-bug/@chrome-bug.

If you need help with this, let me know. We can discuss it here, at the QA mailing list or we can pair on it.

See related thread at QA mailing list:

http://lists.wikimedia.org/pipermail/qa/2013-November/000723.html

Current status with phantomjs 1.9.2 and the latest commit of browsertests repository.

$ export BROWSER_LABEL=phantomjs

$ export MEDIAWIKI_USER=Selenium_user
$ export MEDIAWIKI_PASSWORD=not-the-real-one

$ export MEDIAWIKI_URL=http://commons.wikimedia.beta.wmflabs.org/wiki/
$ bundle exec cucumber -t @commons.wikimedia.beta.wmflabs.org
...
Failing Scenarios:
cucumber features/upload_wizard.feature:28 # Scenario: Navigate to Release rights page
cucumber features/upload_wizard.feature:36 # Scenario: Navigate to Describe page
cucumber features/upload_wizard.feature:46 # Scenario: Navigate to Use page
5 scenarios (3 failed, 2 passed)
37 steps (3 failed, 17 skipped, 17 passed)
4m29.849s

$ export MEDIAWIKI_URL=http://en.wikipedia.beta.wmflabs.org/wiki/
$ bundle exec cucumber -t @en.wikipedia.beta.wmflabs.org
...
Failing Scenarios:
cucumber features/aftv5.feature:22 # Scenario: Click Yes return and click No
cucumber features/preferences_datetime.feature:15 # Scenario: Preferences Date Time
31 scenarios (2 failed, 29 passed)
163 steps (2 failed, 5 skipped, 156 passed)
7m7.580s

$ export MEDIAWIKI_URL=http://test2.wikipedia.org/wiki/
$ bundle exec cucumber -t @test2.wikipedia.org
...
Failing Scenarios:
cucumber features/hotcat.feature:20 # Scenario: Hotcat category link
cucumber features/upload_wizard.feature:23 # Scenario: Navigate to Upload page
cucumber features/upload_wizard.feature:28 # Scenario: Navigate to Release rights page
cucumber features/upload_wizard.feature:36 # Scenario: Navigate to Describe page
cucumber features/upload_wizard.feature:46 # Scenario: Navigate to Use page
39 scenarios (5 failed, 34 passed)
214 steps (5 failed, 21 skipped, 188 passed)
10m58.657s

It is interesting that commons.wikimedia.beta.wmflabs.org failures were Watir::Wait::TimeoutError and Watir::Exception::UnknownObjectException, but commons.wikimedia.beta.wmflabs.org and test2.wikipedia.org were all Net::ReadTimeout (except for one Watir::Exception::UnknownObjectException at test2.wikipedia.org).

If you need more information (and you probably do), feel free to ask questions here, at #wikimedia-qa freenode IRC channel or at QA mailing list:

https://lists.wikimedia.org/mailman/listinfo/qa

Change 101213 had a related patch set uploaded by Zfilipin:
Fix Accept-Language feature[1] for PhantomJS

https://gerrit.wikimedia.org/r/101213

Change 101213 merged by Cmcmahon:
Fix Accept-Language feature[1] for PhantomJS

https://gerrit.wikimedia.org/r/101213

Change 101215 had a related patch set uploaded by Zfilipin:
Fix Accept-Language feature for PhantomJS

https://gerrit.wikimedia.org/r/101215

Change 101215 merged by jenkins-bot:
Fix Accept-Language feature for PhantomJS

https://gerrit.wikimedia.org/r/101215

Change 102099 had a related patch set uploaded by Mayankmadan:
Fixing failing tests on phantomjs

https://gerrit.wikimedia.org/r/102099

Change 102396 had a related patch set uploaded by Mayankmadan:
Fixing Failing tests on PhantomJS

https://gerrit.wikimedia.org/r/102396

Change 102099 abandoned by Mayankmadan:
Fixing failing tests on phantomjs

Reason:
Created a new review request for this bug

https://gerrit.wikimedia.org/r/#/c/102396/

https://gerrit.wikimedia.org/r/102099

Change 102396 abandoned by Mayankmadan:
Fixing Failing tests on PhantomJS

Reason:
There is nothing useful in this commit

https://gerrit.wikimedia.org/r/102396

Change 102634 had a related patch set uploaded by Mayankmadan:
Fix failing tests on phantomjs

https://gerrit.wikimedia.org/r/102634

Change 102664 had a related patch set uploaded by Zfilipin:
Updating Ruby gems

https://gerrit.wikimedia.org/r/102664

Change 102664 merged by jenkins-bot:
Updating Ruby gems

https://gerrit.wikimedia.org/r/102664

jhall wrote:

Only seeing a handful of failures now with PhantomJS 1.9.2:

$ export BROWSER_LABEL=phantomjs
$ export MEDIAWIKI_USER=Selenium_user
$ export MEDIAWIKI_PASSWORD=not-the-real-one
$ export MEDIAWIKI_URL=http://commons.wikimedia.beta.wmflabs.org/wiki/
$ bundle exec cucumber -t @commons.wikimedia.beta.wmflabs.org
Using the default profile...
.............F---...F-----...F---------

(::) failed steps (::)

Net::ReadTimeout (Net::ReadTimeout)
/Users/jeffreyhall/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/net/protocol.rb:152:in `rbuf_fill'
/Users/jeffreyhall/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/net/protocol.rb:134:in `readuntil'
/Users/jeffreyhall/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/net/protocol.rb:144:in `readline'
/Users/jeffreyhall/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/net/http/response.rb:39:in `read_status_line'
/Users/jeffreyhall/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/net/http/response.rb:28:in `read_new'
/Users/jeffreyhall/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/net/http.rb:1406:in `block in transport_request'
/Users/jeffreyhall/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/net/http.rb:1403:in `catch'
/Users/jeffreyhall/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/net/http.rb:1403:in `transport_request'
/Users/jeffreyhall/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/net/http.rb:1376:in `request'
/Users/jeffreyhall/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/net/http.rb:1369:in `block in request'
/Users/jeffreyhall/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/net/http.rb:852:in `start'
/Users/jeffreyhall/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/net/http.rb:1367:in `request'
(eval):1:in `process_watir_call'
./features/step_definitions/upload_wizard_steps.rb:63:in `/^upload file (.+)$/'
features/upload_wizard.feature:30:in `And upload file image.png'

Net::ReadTimeout (Net::ReadTimeout)
/Users/jeffreyhall/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/net/protocol.rb:152:in `rbuf_fill'
/Users/jeffreyhall/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/net/protocol.rb:134:in `readuntil'
/Users/jeffreyhall/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/net/protocol.rb:144:in `readline'
/Users/jeffreyhall/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/net/http/response.rb:39:in `read_status_line'
/Users/jeffreyhall/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/net/http/response.rb:28:in `read_new'
/Users/jeffreyhall/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/net/http.rb:1406:in `block in transport_request'
/Users/jeffreyhall/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/net/http.rb:1403:in `catch'
/Users/jeffreyhall/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/net/http.rb:1403:in `transport_request'
/Users/jeffreyhall/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/net/http.rb:1376:in `request'
/Users/jeffreyhall/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/net/http.rb:1369:in `block in request'
/Users/jeffreyhall/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/net/http.rb:852:in `start'
/Users/jeffreyhall/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/net/http.rb:1367:in `request'
(eval):1:in `process_watir_call'
./features/step_definitions/upload_wizard_steps.rb:63:in `/^upload file (.+)$/'
features/upload_wizard.feature:38:in `And upload file image.png'

Net::ReadTimeout (Net::ReadTimeout)
/Users/jeffreyhall/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/net/protocol.rb:152:in `rbuf_fill'
/Users/jeffreyhall/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/net/protocol.rb:134:in `readuntil'
/Users/jeffreyhall/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/net/protocol.rb:144:in `readline'
/Users/jeffreyhall/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/net/http/response.rb:39:in `read_status_line'
/Users/jeffreyhall/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/net/http/response.rb:28:in `read_new'
/Users/jeffreyhall/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/net/http.rb:1406:in `block in transport_request'
/Users/jeffreyhall/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/net/http.rb:1403:in `catch'
/Users/jeffreyhall/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/net/http.rb:1403:in `transport_request'
/Users/jeffreyhall/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/net/http.rb:1376:in `request'
/Users/jeffreyhall/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/net/http.rb:1369:in `block in request'
/Users/jeffreyhall/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/net/http.rb:852:in `start'
/Users/jeffreyhall/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/net/http.rb:1367:in `request'
(eval):1:in `process_watir_call'
./features/step_definitions/upload_wizard_steps.rb:63:in `/^upload file (.+)$/'
features/upload_wizard.feature:48:in `And upload file image.png'

Failing Scenarios:
cucumber features/upload_wizard.feature:28 # Scenario: Navigate to Release rights page
cucumber features/upload_wizard.feature:36 # Scenario: Navigate to Describe page
cucumber features/upload_wizard.feature:46 # Scenario: Navigate to Use page

5 scenarios (3 failed, 2 passed)
37 steps (3 failed, 17 skipped, 17 passed)
4m27.465s

Change 102634 abandoned by Mayankmadan:
Fix failing tests on phantomjs

https://gerrit.wikimedia.org/r/102634

Change 102634 restored by Mayankmadan:
Fix failing tests on phantomjs

Reason:
All the failing tests are tagged with @phantomjs-bug

https://gerrit.wikimedia.org/r/102634

Change 102634 abandoned by Zfilipin:
Fix failing tests on phantomjs

Reason:
Tests that do not run on phantomjs are now tagged differently. All tests by default have @phantomjs tag, and if the test does not work with the browser, the tag is removed.

https://gerrit.wikimedia.org/r/102634

Resetting status to NEW; no patches left here to review

Željko / Jeff: Do we still have failing tests that a new contributor could fix? Are the instructions provided here still valid?
If both yes, please add "gci2014" to the Whiteboard field of this ticket and potentially "easy" to the Keywords field (if the approach is clear and if this task is self-contained and non-controversial). Thanks!

I am not sure if we care about phantomjs any more at all.

So lets just close this bug. The idea was to easily run browser tests in a headless env since it seemed straight forward to run.

Timo pointed us that PhantomJS is using an outdated Webkit engine that is no more in use by modern browsers. For QUnit tests he is moving to a node module "karma" which uses chromium and an XVFB local server. That runs smoothly.

For our browser tests, we have our bidaily run execute on SauceLabs.

If one day we want to trigger tests on patchset submission, I guess we will use chromium/XVFB as well.