@Ladsgroup let me know if you need help, or if you would like to pair on this.

Change 441051 had a related patch set uploaded (by Zfilipin; owner: Zfilipin):
[mediawiki/core@master] Selenium: run Wikibase repo tests

https://gerrit.wikimedia.org/r/441051

Change 441051 merged by jenkins-bot:
[mediawiki/core@master] Selenium: run Wikibase repo tests

https://gerrit.wikimedia.org/r/441051

Change 441195 had a related patch set uploaded (by Zfilipin; owner: Zfilipin):
[integration/config@master] Run qunit-selenium Jenkins job for Wikibase

https://gerrit.wikimedia.org/r/441195

Change 441195 merged by jenkins-bot:
[integration/config@master] Run qunit-selenium Jenkins job for Wikibase

https://gerrit.wikimedia.org/r/441195

Change 438000 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Start a very basic nodejs browser test

https://gerrit.wikimedia.org/r/438000

On hold until we progress with T226975: [Story] Restructuring Wikibase extension tests

Moving this to the freezer, as one day this will happen, but we long as we can keep running and don't need to actively work on them it is likely not worth the effort.

In T180228#6770168, @Addshore wrote:

Moving this to the freezer, as one day this will happen, but we long as we can keep running and don't need to actively work on them it is likely not worth the effort.

Please, no. This is the last repo to be using this ultra-legacy system which we've been trying to delete for years.

TL;DR: We should take some time to consider alternatives to Selenium+Webdriver.IO before rewriting a large number of Ruby tests.

Reevaluate our choice of browser testing framework

We are about to commit a significant amount of time into rewriting a lot of ruby tests. Before we commit to this massive marathon, we should ask ourselves if the intended goal format, which we have been using for the last 4 years, is still our best option, given our experiences with it so far and how the alternatives developed.
Since the beginning of this year, there were some investigations into Cypress by several WMDE developers, with a generally favorable impression. Also, the Query Builder repository uses Cypress as its browser testing framework.
So if ever consider switching away from Selenium+Webdriver.io, then it should be before we rewrite a large number of tests. Meaning: Now.

Problems with the current setup - reasons to reconsider

1. flakiness: The current tests are sometimes quite flaky, causing unreleated test failures in Jenkins.
   • There is a good chance that part of this not due to Webdriver, but the CI VM doing strange things: T281122
   • OTOH, Lexeme daily tests on beta currently forget that they just clicked the login button and then fail because they can't click the login button

2. deprecation of a central component: We may have to rewrite all Selenium/Webdriver Browser tests to use async/await everywhere at some point:

Caution:
As of 14/04/2021 sync mode will not be supported anymore starting from Node.js v16 due to changes in Chromium. If you start a project with WebdriverIO we recommend to not use sync mode anymore. The community is working on a transition plan, read more on that in this RFC.

• Sync vs. Async Mode

Though maybe there will be a codemod to mass-rewrite all tests. And given the speed at which WMF updates their CI infrastructure, it might be some time till we hit Nodejs 16, so this is probably not an issue for a while. That being said, the new async/await style of writing these tests is really unappealing, because we need to add await to basically everything. How this would look can be seen in a Wikidata Bridge patch.

3. DX: Finally, it seems that the Dev Experience writing browser tests is currently generally low, and it is considered a rather disliked chore. Improving this aspect could improve overall dev satisfaction and help proliferate better development practices such as Outside-In TDD.

Alternatives to Consider

The two main alternatives to be consider would be Cypress and Puppeteer (and maybe Playwright).

• Cypress is already used by the Query Builder project on the WMF CI infrastructure
• Puppeteer is already used by the WMDE FUN team: "dandy" test runner
• (AFAICT, WMDE has no prior experience with Playwright)

WD/WB experience with Cypress

• Cypress was evaluated during a "Ginger Beer and Cake" event in early 2021
• It is used in the Query Builder repository within the WMF CI infrastructure and works with just the standard config

Some Cypress Advantages we noticed

• interactive test runner/dashboard is really nice in terms of DX
• request mocking can be very useful for a frontend project (Though webdriver.io seems to have acquired some capabilities in this regard in one of the v6 minor releases)
• works out-of-the-box with existing WMF CI infrastructure and also locally
• there is some specialized work happening towards enabling first-class vue component testing, based on this VueConfUS 2020 talk

Previous work by WMF:

• Selenium WebdriverIO was chosen as a Ruby alternative in April 2017
  • Puppeteer and Playwright didn't exist back then, Cypress did but apparently wasn't evaluated?
• In Summer 2020 WMF had Google Summer of Code students evaluate Cypress, Puppeteer and Playwright as alternatives to WebdriverIO: Evaluate WebdriverIO replacements for our browser automation framework
  • Playwright Report
  • Puppeteer Report
  • They seemed to have been unable to get Cypress to work. But when I updated Cypress and resubmitted their patch, it work perfectly fine out of the box: Core Cypress test in CI

Next steps

• We need to decide if we want to spend some time evaluating the alternatives
  • We should, because of the reasons mentioned above
• We should write a short ADR on either why we want to continue using WebdriverIO + Selenium, or why decided to go with one of the alternatives

These tests just broke (fixed in a1283ff1b6150bef70c786cd40a060d65c06ad35), but fixing them took me quite some time because I have very little memory of how that stuff works and I have not run them locally in years (and I doubt anyone has!).

Change 921696 had a related patch set uploaded (by Jforrester; author: Jforrester):

[mediawiki/extensions/Wikibase@master] Drop all Ruby browser tests, sadly effectively abandonware

https://gerrit.wikimedia.org/r/921696

Task Review Notes:

• After discussion with Leszek, and in light of upcoming-ui work on the item page, it was concluded that we should be able to remove these tests for the time being. In the case where we want to recreate some of the scenarios covered in these tests, we can check out the commit before removal for a local review.

Task Prio Notes:

• Does not affect end users / production
• Does not affect monitoring
• Affects development efforts
• Does not affect onboarding efforts
• Affects additional stakeholders

Change 921696 merged by jenkins-bot:

[mediawiki/extensions/Wikibase@master] Drop all Ruby browser tests, sadly effectively abandonware

https://gerrit.wikimedia.org/r/921696

Can this task, it's parent (T180451: Port Selenium tests from Ruby to Node.js for WMDE Engineering repositories) and it's subtasks be resolved? There are no more Ruby Selenium tests in Wikibase.

@ItamarWMDE thanks! Is there a reason T180451: Port Selenium tests from Ruby to Node.js for WMDE Engineering repositories and it's subtasks (T222745, T222746, T222748, T222749, T222751) are still open?

For now I'd like to keep them as we go over the wdwb-tech board to see what else is left, and these are good prompts for potentially rewriting the tests these relate to in another technology / framework. But indeed, currently they remain unowned.