Page MenuHomePhabricator

[Spike] Decouple MW-Selenium from Cucumber
Closed, ResolvedPublic

Description

Cucumber provides a high level language abstraction that we've never really utilized for its intended purpose of collaboration,[1] and in most cases the natural language simply adds an unnecessary layer of indirection.[2] The obtuseness of Cucumber's unidiomatic step definition framework also tends to lead to confusion and, I believe, misattribution of the difficulties of writing browser tests to Ruby itself.[3]

I'd like to explore using a lightweight and more idiomatic runner (probably just bare RSpec) to facilitate tests that are "flatter" (implementation alongside description) and therefore likely easier to implement and reason about. MW-Selenium 1.x was heavily refactored to be as modular and runner agnostic as possible. It shouldn't be much work to decouple any remaining bits and implement an RSpec based harness.

At the very least, I think this exercise will help to validate whether it's Ruby or Cucumber that is the biggest barrier to entry (or possibly PageObject, Watir, etc.).

[1] https://cucumber.io/blog/2014/03/03/the-worlds-most-misunderstood-collaboration-tool
[2] https://news.ycombinator.com/item?id=7543994
[3] https://lists.wikimedia.org/pipermail/mobile-l/2015-March/008869.html

Details

Related Gerrit Patches:
mediawiki/selenium : masterBasic RSpec support
mediawiki/selenium : masterDecouple session annotation from Cucumber
mediawiki/selenium : masterDecouple screenshot-ing and artifacts from Cucumber hooks
mediawiki/selenium : masterDecouple check for MediaWiki extension dependencies
mediawiki/selenium : masterDecouple `Environment#test_name`

Event Timeline

dduvall created this task.Aug 7 2015, 12:08 AM
dduvall claimed this task.
dduvall raised the priority of this task from to Medium.
dduvall updated the task description. (Show Details)
dduvall added subscribers: dduvall, zeljkofilipin, greg and 2 others.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 7 2015, 12:08 AM
dduvall updated the task description. (Show Details)Aug 7 2015, 12:08 AM
dduvall set Security to None.
dduvall updated the task description. (Show Details)

Change 230139 had a related patch set uploaded (by Dduvall):
Decouple Environment#test_name

https://gerrit.wikimedia.org/r/230139

Change 230140 had a related patch set uploaded (by Dduvall):
Decouple check for MediaWiki extension dependencies

https://gerrit.wikimedia.org/r/230140

Change 230230 had a related patch set uploaded (by Dduvall):
Decouple screenshot-ing and artifacts from Cucumber hooks

https://gerrit.wikimedia.org/r/230230

Change 230234 had a related patch set uploaded (by Dduvall):
Decouple session annotation from Cucumber

https://gerrit.wikimedia.org/r/230234

Change 230261 had a related patch set uploaded (by Dduvall):
Basic RSpec support

https://gerrit.wikimedia.org/r/230261

dduvall added a comment.EditedAug 8 2015, 1:07 AM

Hurray for a successful spike.

All the core test setup/teardown implementation has been decoupled from Cucumber and a basic harness has been implemented for RSpec—the only feature lacking is the check for MW extension dependencies but the value of those checks have always been a bit dubious anyhow.

From here, we should probably do some experimentation with teams/individuals writing browser tests. @Jdlrobson, I think your feedback on this alternative form would be tremendously valuable. Perhaps would could pair on it soon.

As an example of how this new form would look, the following is a port of some of the Cucumber scenarios from core's login.feature (this is the whole shebang, there are no separate description/implementation files):

Using the standard RSpec DSL:

describe 'login page' do
  describe 'login form' do
    it 'includes username/password fields, and a login button' do
      visit(LoginPage) do |page|
        expect(page.username_element).to be_visible
        expect(page.password_element).to be_visible
        expect(page.login_button_element).to be_visible
      end
    end

    it 'shows an error when logging in without credentials' do
      visit(LoginPage) do |page|
        page.login_element.click

        expect(page.error_box_element).to be_visible
      end
    end

    it 'shows an error when logging in with bad credentials' do
      visit(LoginPage) do |page|
        page.login_with('some bad username', 'some bad password')

        expect(page.error_box_element).to be_visible
      end
    end
  end
end

Another option is to use feature/scenario syntax (this isn't native to RSpec, but implemented by simply aliasing describe as feature, before as background, and it as scenario):

feature 'Logging in' do
  scenario 'Form contains username/password fields, and a login button' do
    visit(LoginPage) do |page|
      expect(page.username_element).to be_visible
      expect(page.password_element).to be_visible
      expect(page.login_button_element).to be_visible
    end
  end

  scenario 'Logging in without credentials shows an error' do
    visit(LoginPage) do |page|
      page.login_element.click

      expect(page.error_box_element).to be_visible
    end
  end

  scenario 'Logging in with bad credentials shows an error' do
    visit(LoginPage) do |page|
      page.login_with('some bad username', 'some bad password')

      expect(page.error_box_element).to be_visible
    end
  end
end

The latter form is closer to what we do now with Cucumber, but I personally prefer the first form as I think it promotes more descriptive examples—perhaps this is just because I'm used to writing it.

We'll also want to separate out the Cucumber and RSpec harnesses into separate gems: Fewer dependencies for each and fewer dependency conflicts (Cucumber 1.x forces us to still use RSpec 2.x).

Added this task to Testing Initiative 2015. It isn't quite "using another language" for browser testing, but it does constitute a different form/syntax for tests in Ruby.

dduvall moved this task from Backlog to Radar on the Testing-Initiative board.Aug 12 2015, 9:36 PM

Change 230139 merged by jenkins-bot:
Decouple Environment#test_name

https://gerrit.wikimedia.org/r/230139

Change 230140 merged by jenkins-bot:
Decouple check for MediaWiki extension dependencies

https://gerrit.wikimedia.org/r/230140

Change 230230 merged by jenkins-bot:
Decouple screenshot-ing and artifacts from Cucumber hooks

https://gerrit.wikimedia.org/r/230230

Change 230234 merged by jenkins-bot:
Decouple session annotation from Cucumber

https://gerrit.wikimedia.org/r/230234

Change 230261 merged by jenkins-bot:
Basic RSpec support

https://gerrit.wikimedia.org/r/230261

dduvall closed this task as Resolved.Aug 25 2015, 3:18 PM