Page MenuHomePhabricator

[LEX] [TECH] Decide on a way forward with browser testing
Closed, ResolvedPublic

Description

As of change 944969 browser testing has been disabled on WikibaseLexeme due to incompatibilities with WebdriverIO's sync mode. To ensure and improve the testability of the extension, we should choose between two possible routes:

  1. Write new Selenium/Webdriver tests in the async format and rewrite existing files incrementally to async
  2. Write new Cypress browser tests and rewrite existing files incrementally to Cypress

With this task, we will investigate the qualities and trade-offs of the two approaches above. The investigation should provide a concise description of each approach and by guided by the following questions:

  • What setup costs does each approach incur?
  • How compatible is each approach with our current CI?

- How performant is each approach, both locally and on CI servers? Answered in T331732#8703102

  • How does the chosen approach impact onboarding efforts for new developers?

- How documented is each approach to support possible future troubleshooting and development efforts? Answered in T331732#8705071

If there are any additional comparison points, please feel free to add them to the investigation report.

Acceptance Criteria

  • An investigation report is made available for the team, detailing the above information.
  • A decision is made, and an EPIC for the migration process is created.

Event Timeline

Task Review:

  • This is a spike task to try and decided on a way forward with browser testing in WikibaseLexeme.

Prio Notes:

  • Does not affect end users / production
  • Affects monitoring
  • Affect development efforts
  • Might affect onboarding efforts
  • Might affect additional stakeholders (potentially, these tests are run in bigger settings than just the Wikidata team, and failures / blocked dependencies can cascade)
ItamarWMDE renamed this task from [LEX] Decide on a way forward with browser testing to [SW] [LEX] [TECH] Decide on a way forward with browser testing.Oct 10 2023, 1:59 PM
ItamarWMDE moved this task from [DOT] By Project to [DOT] Prioritized on the wmde-wikidata-tech board.
ItamarWMDE renamed this task from [SW] [LEX] [TECH] Decide on a way forward with browser testing to [LEX] [TECH] Decide on a way forward with browser testing.Oct 11 2023, 10:43 AM

@ArthurTaylor and I talked about this task this morning, and given that we recently decided for Cypress in EntitySchema (T331732) and don’t see any reason to decide differently here, we think that we should also pick Cypress in WikibaseLexeme and other codebases maintained by us.

What setup costs does each approach incur?

Nothing too bad, probably – T333427 (Cypress setup for EntitySchema) seems to have gone fairly well.

How compatible is each approach with our current CI?

Apparently both approaches are equally compatible, given that we have instances of both running in CI at the moment. (According to T331732#8705071, even “capturing video and meaningful logs” “works out of the box” with Cypress, which is more than I would’ve expected tbh.)

How does the chosen approach impact onboarding efforts for new developers?

Anecdotally, our newest developer, Arthur, is familiar with Cypress and not with WebdriverIO; given their general usage statistics (Cypress seems to be more popular than wdio), I think we can expect this to be likely for future hires as well.

Unrelatedly, I also learned today that WMDE-TechWish are migrating Cite to Cypress: T353436; notably, in contrast to our own experience in T331732#8703102, they report that Cypress seems to run much faster in CI.

Yep - agreed. I think if other teams are going for Cypress, we already have Cypress tests, we're all familiar with Cypress, and it's an active project with a large user base, we can't go too far wrong here.

We should probably still write an ADR for WikibaseLexeme?

Probably makes sense. What is the process for ADRs? Is there a template?

There’s a brief explanation of ADRs in WikibaseLexeme ADR 0001; Wikibase has a short template in docs/adr/NNNN-adr-template.md.

Change 993082 had a related patch set uploaded (by Lucas Werkmeister (WMDE); author: Lucas Werkmeister (WMDE)):

[mediawiki/extensions/WikibaseLexeme@master] Add ADR for Cypress browser testing decision

https://gerrit.wikimedia.org/r/993082

Change 993082 merged by jenkins-bot:

[mediawiki/extensions/WikibaseLexeme@master] Add ADR for Cypress browser testing decision

https://gerrit.wikimedia.org/r/993082