Page MenuHomePhabricator

Port Wikibase Selenium tests from Ruby to Node.js
Open, LowPublic

Description

As a wikibase developer I want to use the nodejs test framework so that I only have 1 selenium test framework to think about.
As the WMF I want to use the nodejs test framework in wikibaselexeme so that all wmf deployed extensions are consistent.
As release engineering I want to remove all ruby selenium jobs so that more CI time is available for other jobs.

Details

  1. Jenkins jobs to setup
    • mediawiki/core: master: Selenium: run Wikibase repo tests 441051
    • integration/config: master: Run qunit-selenium Jenkins job for Wikibase 441195
    • mediawiki/extensions/Wikibase: master: Start a very basic nodejs browser test 438000
  2. Features to migrate

note: keep in mind that if some of those feature tests does not make sense then we would just drop it and do not have to migrate it just for the sake of it ;)

  • aliases.feature
  • authority_control_gadget.feature
  • delete_item.feature
  • description.feature
  • edit_tab.feature
  • empty_label_and_description.feature
  • header.feature
  • item_smoke.feature
  • label.feature
  • non_existing_item.feature
  • property_smoke.feature
  • reference.feature
  • sitelinks.feature
  • sitelinks_badge.feature
  • special_set_label.feature
  • statement.feature
  • statements_geo.feature
  • statements_item.feature
  • statements_monolingual.feature
  • statements_quantity.feature
  • statements_rank.feature
  • statements_snaktype.feature
  • statements_string.feature
  • statements_time.feature
  • statements_url.feature

Impact & Priority

There is no specific point in time when the migration should happen (i.e. does not have happen immediately).
The WMDE is no longer supporting the ruby framework so if it does break we will be on our own and then forced to make fixes / migrate before Wikibase development can continue.
The ruby tests do not run as part of the mediawiki core gate-submit jobs, although the node ones will. Having our tests run with the core gets submit tests will eliminate breakages caused by core changes.

Task

Convert all ruby features to be run as part of the node JS selenium tests.
A test framework example can probably be taken from WikibaseLexeme which already includes node browser tests.

Acceptance Criteria

  • No ruby test features exist in the Wikibase git repo code base
  • All features that did exist have been converted to the node js system
  • The "mwext-mw-selenium-composer-jessie" jenkins job that runs on Wikibase git code changes in gate-submit no longer runs on them
  • All other files & docs relating to the ruby tests have been removed from the Wikibase git repo

Related Objects

Event Timeline

Change 438000 had a related patch set uploaded (by Ladsgroup; owner: Amir Sarabadani):
[mediawiki/extensions/Wikibase@master] Start a very basic nodejs browser test

https://gerrit.wikimedia.org/r/438000

@Ladsgroup let me know if you need help, or if you would like to pair on this.

Change 441051 had a related patch set uploaded (by Zfilipin; owner: Zfilipin):
[mediawiki/core@master] Selenium: run Wikibase repo tests

https://gerrit.wikimedia.org/r/441051

Change 441051 merged by jenkins-bot:
[mediawiki/core@master] Selenium: run Wikibase repo tests

https://gerrit.wikimedia.org/r/441051

Change 441195 had a related patch set uploaded (by Zfilipin; owner: Zfilipin):
[integration/config@master] Run qunit-selenium Jenkins job for Wikibase

https://gerrit.wikimedia.org/r/441195

Change 441195 merged by jenkins-bot:
[integration/config@master] Run qunit-selenium Jenkins job for Wikibase

https://gerrit.wikimedia.org/r/441195

Change 438000 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Start a very basic nodejs browser test

https://gerrit.wikimedia.org/r/438000

Addshore renamed this task from Port Wikibase Selenium tests from Ruby to Node.js to Migrate browser tests of Wikibase to nodejs framework.Aug 19 2018, 11:49 AM
Addshore triaged this task as Low priority.
Addshore updated the task description. (Show Details)
Addshore moved this task from incoming to ready to go on the Wikidata board.
Addshore moved this task from Inbox to Goals on the wdwb-tech board.
Addshore renamed this task from Migrate browser tests of Wikibase to nodejs framework to Port Wikibase Selenium tests from Ruby to Node.js.Aug 19 2018, 11:51 AM
alaa_wmde raised the priority of this task from Low to Medium.Apr 23 2019, 9:15 AM
alaa_wmde updated the task description. (Show Details)
alaa_wmde moved this task from Incoming to Ready to estimate on the Wikidata-Campsite board.

Moving this to the freezer, as one day this will happen, but we long as we can keep running and don't need to actively work on them it is likely not worth the effort.

Moving this to the freezer, as one day this will happen, but we long as we can keep running and don't need to actively work on them it is likely not worth the effort.

Please, no. This is the last repo to be using this ultra-legacy system which we've been trying to delete for years.

TL;DR: We should take some time to consider alternatives to Selenium+Webdriver.IO before rewriting a large number of Ruby tests.

Reevaluate our choice of browser testing framework

We are about to commit a significant amount of time into rewriting a lot of ruby tests. Before we commit to this massive marathon, we should ask ourselves if the intended goal format, which we have been using for the last 4 years, is still our best option, given our experiences with it so far and how the alternatives developed.
Since the beginning of this year, there were some investigations into Cypress by several WMDE developers, with a generally favorable impression. Also, the Query Builder repository uses Cypress as its browser testing framework.
So if ever consider switching away from Selenium+Webdriver.io, then it should be before we rewrite a large number of tests. Meaning: Now.

Problems with the current setup - reasons to reconsider
  1. flakiness: The current tests are sometimes quite flaky, causing unreleated test failures in Jenkins.
  1. deprecation of a central component: We may have to rewrite all Selenium/Webdriver Browser tests to use async/await everywhere at some point:

Caution:
As of 14/04/2021 sync mode will not be supported anymore starting from Node.js v16 due to changes in Chromium. If you start a project with WebdriverIO we recommend to not use sync mode anymore. The community is working on a transition plan, read more on that in this RFC.

Though maybe there will be a codemod to mass-rewrite all tests. And given the speed at which WMF updates their CI infrastructure, it might be some time till we hit Nodejs 16, so this is probably not an issue for a while. That being said, the new async/await style of writing these tests is really unappealing, because we need to add await to basically everything. How this would look can be seen in a Wikidata Bridge patch.

  1. DX: Finally, it seems that the Dev Experience writing browser tests is currently generally low, and it is considered a rather disliked chore. Improving this aspect could improve overall dev satisfaction and help proliferate better development practices such as Outside-In TDD.
Alternatives to Consider

The two main alternatives to be consider would be Cypress and Puppeteer (and maybe Playwright).

  • Cypress is already used by the Query Builder project on the WMF CI infrastructure
  • Puppeteer is already used by the WMDE FUN team: "dandy" test runner
  • (AFAICT, WMDE has no prior experience with Playwright)
WD/WB experience with Cypress
Some Cypress Advantages we noticed
  • interactive test runner/dashboard is really nice in terms of DX
  • request mocking can be very useful for a frontend project (Though webdriver.io seems to have acquired some capabilities in this regard in one of the v6 minor releases)
  • works out-of-the-box with existing WMF CI infrastructure and also locally
  • there is some specialized work happening towards enabling first-class vue component testing, based on this VueConfUS 2020 talk
Previous work by WMF:
Next steps
  • We need to decide if we want to spend some time evaluating the alternatives
    • We should, because of the reasons mentioned above
  • We should write a short ADR on either why we want to continue using WebdriverIO + Selenium, or why decided to go with one of the alternatives
Addshore lowered the priority of this task from Medium to Low.Jul 14 2021, 11:13 AM