To ease their writing, I propose the use of:

• Test scenarios written in Gherkin syntax thanks to Cucumber;
• Playwright in order to manipulate the headless browsers (Firefox, Chrome, ... but not IE);

Scenarios written in Gherkin would allow to follow a Behavior-Driven Development, so when the team agrees to implement a specific feature, everyone can participate to the writing on the scenario before the implementation.

An example of 2 scenarios:

Feature: Guess the word

  # The first example has two steps
  Scenario: Maker starts a game
    When the Maker starts a game
    Then the Maker waits for a Breaker to join

  # The second example has three steps
  Scenario: Breaker joins a game
    Given the Maker has started a game with the word "silky"
    When the Breaker joins the Maker's game
    Then the Breaker must guess a word with 5 characters

Then when the team agrees on a scenario, Cucumber allows to parse the scenario and for each sentence. See this example (in Java): https://cucumber.io/docs/guides/browser-automation/?lang=java

Playwright would be used instead of Selenium, because it seems to be faster and to propose an easier API. Although Selenium can be a good fit if you're willing to test on every browsers (however that would require a much more complex installation).

I can try to implement a prototype if you want to.

Some links:

• Cucumber documentation: https://cucumber.io/docs/guides/10-minute-tutorial/?lang=java
• E2E testing with Cucumber and Playwright: https://tally-b.medium.com/e2e-testing-with-cucumber-and-playwright-9584d3ef3360

Hey Teleosteen, thanks for making this ticket. It seems like it's probably a duplicate of https://phabricator.wikimedia.org/T276847. I'm very interested in getting more robust end-to-end tests. For what it's worth, Mediawiki mostly uses Selenium for browser based testing (see https://www.mediawiki.org/wiki/Selenium), and we already have a couple (literally two) basic end-to-end tests the are run on patch merge requests.

We have seven "critical user journeys" that we're interested in covering with end-to-end tests:

1. evaluate a function
2. create a function definition
3. edit a function definition
4. create tests
5. create implementations (via code or composition)
6. connect an implementation or test to a function
7. create and edit types

We should probably have tickets for these (or rather, split up this ticket: https://phabricator.wikimedia.org/T276847). Would you be interested in writing some of these tests? If so, I'll split out some tickets and can assign them to you.