Page MenuHomePhabricator

Setup a framework for e2e test automation
Closed, ResolvedPublic16 Estimated Story Points

Description

Setup a framework for SX / CX e2e test automation.

Current plan is to use Webdriver.io (https://webdriver.io/)

Current status

See: https://phabricator.wikimedia.org/T312543#9586537

  • Setup / choose the base / framework for the e2e tests - Webdriver.io @EChukwukere-WMF / @abi_
  • Identify and write a few basic tests for CX - @abi_
    • Need to update tests to use HTTP interceptors
  • Identify and write a few basic tests for SX - @ngkountas
    • To ensure that we can test the section translation workflow with webdriver.io
  • Ensure that developers are able to run the tests locally.
    • Emeka, Nik and I are able to run the tests locally.
  • Setup an HTML reporter for showing test results @abi_
  • Either setup e2e run on CI or on a periodic basis somewhere else
  • Send test reports via email or ensure that they are available on a public URL.

Last updated on 2024-02-29


Previous description:

Set up Cypress to automate content translation workflows as well as SX workflows. The initial phase of this work is to config the tool and include this as part of our repo.

Documentation to get started with: https://docs.cypress.io/guides/overview/why-cypress#What-you-ll-learn

After set up further discussion will be needed to prioritize the workflow to automate for a start

Event Timeline

Pginer-WMF triaged this task as Medium priority.Aug 1 2022, 11:28 AM
EChukwukere-WMF added a subscriber: KartikMistry.

I am blocked on this because of ann error I am observing during contentTranslation extention set up on local. @KartikMistry is helping me investigate this

I am blocked on this because of ann error I am observing during contentTranslation extention set up on local. @KartikMistry is helping me investigate this

Meanwhile would it be feasible to point to the test Wikipedia instance for the mobile and desktop experiences?

@Pginer-WMF yes it is. I do already know how to configure that with cypress. you gave me both links. I just need to get over a local set up blocker with contentTranslation extention set up. I will provide feedback by next week

Change 840187 had a related patch set uploaded (by Echukwukere; author: Echukwukere):

[mediawiki/extensions/ContentTranslation@master] Cypress test framework

https://gerrit.wikimedia.org/r/840187

All,

I will be making a wholesome change to this framework. I did consult with @zeljkofilipin who recommended webdriverIO ( this also uses javascript). The reason been that alot of test framework projects already use this framework should in case I need community help within the org and in cases others will like to contribute to it it will be a seamless transition. I did take a few days to learn about this framework on udemy. In a couple days I should make this change and update the patch.

Let me know if anyone has any objections to this.

thanks

EChukwukere-WMF renamed this task from Set up Cypress framework for Automation to Set up framework for Automation.Nov 29 2022, 3:25 AM

I was checking in Codesearch what other teams use webdriver.io, and it seems its used in quite a few places at the Foundation. One of the prominent users being WikiLambda: https://github.com/wikimedia/mediawiki-extensions-WikiLambda/tree/master/tests/selenium; I think we should take inspiration from that code base to write our e2e code for CX / SX.

They also appear to be maintaining the tests under the main extension Gerrit repository, rather than having a separate repository for testing. I think this is a good idea as it makes it easier to update tests in case of code changes.

The deploy and test pipelines for WikiLambda appear to be run on GitLab via a bot: AW GitLab Bot and a separate repository on GitLab: https://gitlab.wikimedia.org/repos/abstract-wiki/aw-e2e. Here's a sample patch on Gerrit: 968336: ZTypedListItems: Add a tooltip & aria-label to the + button for accessibility (and to stop Vue whining)

Change 971557 had a related patch set uploaded (by Abijeet Patro; author: Abijeet Patro):

[mediawiki/extensions/ContentTranslation@master] e2e: Add webdriver.io

https://gerrit.wikimedia.org/r/971557

Change 971558 had a related patch set uploaded (by Abijeet Patro; author: Abijeet Patro):

[mediawiki/extensions/ContentTranslation@master] e2e: Update tests to follow Wikimedia guidelines

https://gerrit.wikimedia.org/r/971558

Change 971618 had a related patch set uploaded (by Abijeet Patro; author: Abijeet Patro):

[mediawiki/extensions/ContentTranslation@master] e2e: Add tests for suggestions in the unified dashboard

https://gerrit.wikimedia.org/r/971618

Change 972538 had a related patch set uploaded (by Abijeet Patro; author: Abijeet Patro):

[mediawiki/extensions/ContentTranslation@master] e2e: Add test to dismiss suggestion in the unified dashboard

https://gerrit.wikimedia.org/r/972538

Change 972539 had a related patch set uploaded (by Abijeet Patro; author: Abijeet Patro):

[mediawiki/extensions/ContentTranslation@master] e2e: Add test to refresh suggestions

https://gerrit.wikimedia.org/r/972539

Change 973416 had a related patch set uploaded (by Abijeet Patro; author: Abijeet Patro):

[mediawiki/extensions/ContentTranslation@master] e2e: Update test description for unified dashboard

https://gerrit.wikimedia.org/r/973416

Change 973417 had a related patch set uploaded (by Abijeet Patro; author: Abijeet Patro):

[mediawiki/extensions/ContentTranslation@master] e2e: Add test to start and save a new translation on desktop

https://gerrit.wikimedia.org/r/973417

abi_ changed the task status from Open to In Progress.Nov 23 2023, 12:34 PM
abi_ set the point value for this task to 16.

Change 971557 merged by jenkins-bot:

[mediawiki/extensions/ContentTranslation@master] e2e: Add webdriver.io

https://gerrit.wikimedia.org/r/971557

Change 971558 merged by jenkins-bot:

[mediawiki/extensions/ContentTranslation@master] e2e: Update tests to follow Wikimedia guidelines

https://gerrit.wikimedia.org/r/971558

Change 971618 merged by jenkins-bot:

[mediawiki/extensions/ContentTranslation@master] e2e: Add tests to (un)favorite suggestions in the unified dashboard

https://gerrit.wikimedia.org/r/971618

Change 976704 had a related patch set uploaded (by Nik Gkountas; author: Nik Gkountas):

[mediawiki/extensions/ContentTranslation@master] Add wdio intercept service and use it in e2e tests

https://gerrit.wikimedia.org/r/976704

Change 976704 merged by jenkins-bot:

[mediawiki/extensions/ContentTranslation@master] Add wdio intercept service and use it in e2e tests

https://gerrit.wikimedia.org/r/976704

Change 972538 merged by jenkins-bot:

[mediawiki/extensions/ContentTranslation@master] e2e: Add test to dismiss suggestion in the unified dashboard

https://gerrit.wikimedia.org/r/972538

Change 972539 merged by jenkins-bot:

[mediawiki/extensions/ContentTranslation@master] e2e: Add test to refresh suggestions

https://gerrit.wikimedia.org/r/972539

Change 973416 merged by jenkins-bot:

[mediawiki/extensions/ContentTranslation@master] e2e: Update test description for unified dashboard

https://gerrit.wikimedia.org/r/973416

abi_ added a subscriber: ngkountas.
abi_ renamed this task from Set up framework for Automation to Setup a framework for e2e test automation.Nov 30 2023, 12:17 PM
abi_ updated the task description. (Show Details)

Change 980081 had a related patch set uploaded (by Abijeet Patro; author: Abijeet Patro):

[mediawiki/extensions/ContentTranslation@master] Use WDIO interceptor when dismissing a suggestion

https://gerrit.wikimedia.org/r/980081

Change 980085 had a related patch set uploaded (by Abijeet Patro; author: Abijeet Patro):

[mediawiki/extensions/ContentTranslation@master] Rename BrowserHelper to InterceptorService

https://gerrit.wikimedia.org/r/980085

Change 980086 had a related patch set uploaded (by Abijeet Patro; author: Abijeet Patro):

[mediawiki/extensions/ContentTranslation@master] Use WDIO interceptor when refreshing suggestions

https://gerrit.wikimedia.org/r/980086

Change 981437 had a related patch set uploaded (by Abijeet Patro; author: Abijeet Patro):

[mediawiki/extensions/ContentTranslation@master] Fix ESLint formatting warnings about line length

https://gerrit.wikimedia.org/r/981437

Change 981438 had a related patch set uploaded (by Abijeet Patro; author: Abijeet Patro):

[mediawiki/extensions/ContentTranslation@master] Use dotenv library to load variables in .env file

https://gerrit.wikimedia.org/r/981438

Change 980081 merged by jenkins-bot:

[mediawiki/extensions/ContentTranslation@master] Use WDIO interceptor when dismissing a suggestion

https://gerrit.wikimedia.org/r/980081

Change 980085 merged by jenkins-bot:

[mediawiki/extensions/ContentTranslation@master] Rename BrowserHelper to InterceptorService

https://gerrit.wikimedia.org/r/980085

Change 980086 merged by jenkins-bot:

[mediawiki/extensions/ContentTranslation@master] Use WDIO interceptor when refreshing suggestions

https://gerrit.wikimedia.org/r/980086

Change 981437 merged by jenkins-bot:

[mediawiki/extensions/ContentTranslation@master] Fix ESLint formatting warnings about line length

https://gerrit.wikimedia.org/r/981437

Change 981438 merged by Nik Gkountas:

[mediawiki/extensions/ContentTranslation@master] Use dotenv library to load variables in .env file

https://gerrit.wikimedia.org/r/981438

Change 973417 merged by jenkins-bot:

[mediawiki/extensions/ContentTranslation@master] e2e: Add test to start and save a new translation on desktop

https://gerrit.wikimedia.org/r/973417

Marking this task as done. As part of the work for this task, we've identified that we will be using webdriver.io as the framework to use for our end-to-end tests. We've used it to write a few tests that are being run manually whenever a new SX build is created.

We'll create separate tasks based on discussions that we have within the team on the scope of the e2e tests.