Page MenuHomePhabricator

Proposal: end-to-end tests with Gherkin syntax and automated browsers
Closed, DuplicatePublic

Description

End-to-end tests would help to ensure that the whole system works as expected for most of the cases, and that every component integrates and interacts well altogether.

They are more complex than unit tests or integration tests, so they should be fewer.

Event Timeline

To ease their writing, I propose the use of:

  • Test scenarios written in Gherkin syntax thanks to Cucumber;
  • Playwright in order to manipulate the headless browsers (Firefox, Chrome, ... but not IE);

Scenarios written in Gherkin would allow to follow a Behavior-Driven Development, so when the team agrees to implement a specific feature, everyone can participate to the writing on the scenario before the implementation.

An example of 2 scenarios:

Feature: Guess the word

  # The first example has two steps
  Scenario: Maker starts a game
    When the Maker starts a game
    Then the Maker waits for a Breaker to join

  # The second example has three steps
  Scenario: Breaker joins a game
    Given the Maker has started a game with the word "silky"
    When the Breaker joins the Maker's game
    Then the Breaker must guess a word with 5 characters

Then when the team agrees on a scenario, Cucumber allows to parse the scenario and for each sentence. See this example (in Java): https://cucumber.io/docs/guides/browser-automation/?lang=java

Playwright would be used instead of Selenium, because it seems to be faster and to propose an easier API. Although Selenium can be a good fit if you're willing to test on every browsers (however that would require a much more complex installation).

I can try to implement a prototype if you want to.


Some links:

Hey Teleosteen, thanks for making this ticket. It seems like it's probably a duplicate of https://phabricator.wikimedia.org/T276847. I'm very interested in getting more robust end-to-end tests. For what it's worth, Mediawiki mostly uses Selenium for browser based testing (see https://www.mediawiki.org/wiki/Selenium), and we already have a couple (literally two) basic end-to-end tests the are run on patch merge requests.

We have seven "critical user journeys" that we're interested in covering with end-to-end tests:

  1. evaluate a function
  2. create a function definition
  3. edit a function definition
  4. create tests
  5. create implementations (via code or composition)
  6. connect an implementation or test to a function
  7. create and edit types

We should probably have tickets for these (or rather, split up this ticket: https://phabricator.wikimedia.org/T276847). Would you be interested in writing some of these tests? If so, I'll split out some tickets and can assign them to you.