Page MenuHomePhabricator

[ES-M1] Investigate browser testing frameworks for Entity Schema Extension
Closed, ResolvedPublic

Description

Currently, we are employing a rather old (written in legacy synchronous style) and flaky small set of selenium browser tests in the EntitySchema extension.

To improve the testability of the extension, we should choose between two possible routes:

  1. Write new Selenium/Webdriver tests in the async format and rewrite existing files incrementally to async
  2. Write new Cypress browser tests and rewrite existing files incrementally to Cypress

With this task, we will investigate the qualities and trade-offs of the two approaches above. The investigation should provide a concise description of each approach and by guided by the following questions:

  • What setup costs does each approach incur?
  • How compatible is each approach with our current CI?
  • How performant is each approach, both locally and on CI servers?
  • How does the chosen approach impact onboarding efforts for new developers?
  • How documented is each approach to support possible future troubleshooting and development efforts?

If there are any additional comparison points, please feel free to add them to the investigation report.

Acceptance Criteria

  • An investigation report is made available for the team, detailing the above information.

Event Timeline

Prio Notes:

  • Impact Areas: Testability
  • Does not affect production / end users
  • Does affect development efforts
  • Does not affect onboarding efforts (but should be considered during investigation)
  • Does not affect additional stakeholders
ItamarWMDE renamed this task from [ES-M2] Investigate browser testing frameworks for Entity Schema Extension to [ES-M1] Investigate browser testing frameworks for Entity Schema Extension.Mar 14 2023, 2:19 PM
ItamarWMDE added a project: Wikidata Dev Team.

The task was moved to MS 1 as it blocks from writing new browser tests

Unscientific duration measurement (lots of other stuff running in the background, only two measurements each).

Also, all this is just for the HelloWorld and ViewSchema test files.

baseline (wdio7, node 14 via nvm): 20.665 seconds, 20.823 seconds
baseline (wdio7, node 14 via mwcli fresh, headless): 24.432 seconds, 19.076 seconds

async (wdio7, node 16 via nvm): 21.629 seconds, 26.396 seconds (it had some problems with a second tab opening with the "url" being "http://window-size%3D1920%2C1080/", this can probably be fixed)
async (wdio7, node 16 via mwcli fresh, headless): 17.291 seconds, 17.674 seconds

cypress (node 16 via mwcli fresh, headless): 38.531 seconds, 33.091 seconds

I notice that there seems to be substantial "setup cost" which might have undue wait in a scenario that only consists of two small test files. Thus, the difference might not be as stark as it seems here. Still, Cypress appears to be consistently somewhat slower.

Async WebdriverCypress
What setup costs does each approach incur?We need to rewrite all tests from hand to add async/await or Promises. While there is a codemod, it does more harm than good. However, we can make use of the code and the patterns that already exist. Though, we will have to invest some extra time to fix the existing flakyness.We have to write all tests from scratch. But doing that already, I found it quite a nice thing to do for those two tests. We may or may not introduce new flakiness with the rewrite that we have to look into. But there is hope that the way to write Cypress tests results in less flakiness by default.
How compatible is each approach with our current CI?Very, it might even support runnning tests of a single suite in parallel? T226869Running Cypress tests in CI works out of the box, including capturing video and meaningful logs. However, running the tests of an Extension like EntitySchema in parallel is maybe not possible, because it requires orchestration by an external cypress.io service and that might not work with our bespoke CI setup. It might be possible with GitLab CI though, once we migrated there.
How performant is each approach, both locally and on CI servers?Two simple tests in CI: 18 secondsTwo simple tests in CI: 39 seconds
How does the chosen approach impact onboarding efforts for new developers?Pausing a test to interact with a page is always a hazzle. Window closes after test is done. No "interactive" way to run tests.Writing new tests is easy, because Cypress is keeps the window open with dangling state to allow for interacting with the system after the test finished.
How documented is each approach to support possible future troubleshooting and development efforts?Webdriver.io has comprehensive documentation on their website as well. Though mainly on their website, there seems to be little content around it. Selenium documentation is often more confusing than helping, because many examples and references are in Java and other languages.Cypress has a comprehensive documentation, with lots of content being produced around OSS events and frontend frameworks and by the community. However, as usual, community content can be great, but it can also be sometimes an inconsistent mixed bag

My main takeaways:

  • Async Webdriver.io is faster. (Probably not the factor 2 from the numbers above, but noticable.)
  • Cypress runs in the browser, and there are some trade-offs because of that. One thing that might affect us, is that we cannot use multiple tabs. But I think we can work around that limitation in the edge cases where we might want it in the first place
  • Cypress is much nicer to work with, write and debug
  • documentation is good for both, somewhat with an edge for Cypress
  • It seems easier to write Cypress test that are not flaky, though some flakyness can happen there as well
  • Cypress tests are plain nicer to write, especially compared to new selenium tests that need an await in front of everything
  • Cypress should be easier to onboard people to, both due to the previous point and because it is much more popular:
    image.png (488×631 px, 25 KB)

Change 900638 had a related patch set uploaded (by Michael Große; author: Michael Große):

[mediawiki/extensions/EntitySchema@master] [PoC] Add cypress for two simple tests

https://gerrit.wikimedia.org/r/900638

Change 900420 had a related patch set uploaded (by Michael Große; author: Michael Große):

[mediawiki/extensions/EntitySchema@master] [PoC] Migrate two simple tests to async selenium/wdio

https://gerrit.wikimedia.org/r/900420

👍 Thank you for the comprehensive report! I will prepare some way to make this decision once it is out of review (hopefully before next sprint)

I've recently stumbled upon The Evolution of Browser Automation. I think it's a good overview of browser automation tools.

I've recently stumbled upon The Evolution of Browser Automation. I think it's a good overview of browser automation tools.

That was quite interesting to watch, thank you!

ItamarWMDE updated the task description. (Show Details)

Complete and we made a decision to continue with cypress. See T333427: [ES-M2] Setup Cypress for browser testing for setup task.

Change 900638 abandoned by Michael Große:

[mediawiki/extensions/EntitySchema@master] [PoC] Add cypress for two simple tests

Reason:

Superseded by Ifd127298f

https://gerrit.wikimedia.org/r/900638

Change 900420 abandoned by Michael Große:

[mediawiki/extensions/EntitySchema@master] [PoC] Migrate two simple tests to async selenium/wdio

Reason:

We decided to go the Cypress route

https://gerrit.wikimedia.org/r/900420