Page MenuHomePhabricator

[Spike] Have a discussion around Minerva selenium browser test architecture
Closed, ResolvedPublic0 Estimated Story Points

Description

In https://gerrit.wikimedia.org/r/425422 concerns were raised by @zeljkofilipin about the readability of Minervas browser tests.

In T219920 given our browser tests disappeared we prioritised getting them running again.

With the dust settled on that urgency we should consider what changes are needed to bring consistency to the browser tests compared to other extensions.

In particular:

  1. Do we still want to use cucumber long term? if so what is the motivation ?
  1. if not, let's remove the feature files inside the new tests. What improvements can be made to the browser tests that promote readability but also reusability. How best to strike a balance between the two?

proposal

I suggest we meet and chat through the code and existing patterns and find a compromise we are happy with.

Event Timeline

If we want to meet soon, it has to be in the next few days. I'll be traveling a bit this month. I've sent calendar invite to @Jdlrobson for Monday April 15 at 5:30pm UTC. Let me know if that's not a good time or if I need to invite more people. You should be able to edit the event yourself.

Jdlrobson moved this task from Incoming to Needs Prioritization on the Readers-Web-Backlog board.
Jdlrobson renamed this task from Have a discussion around Minerva selenium browser test architecture to [Spike] Have a discussion around Minerva selenium browser test architecture.May 1 2019, 9:05 PM
Jdlrobson claimed this task.
Jdlrobson added a project: Spike.
Jdlrobson set the point value for this task to 0.

(conversation happening tomorrow and will report back)

Things we discussed today:

  1. Reading web is stuck on T221860. Not sure how to proceed with getting that patch merged since cannot reproduce. Maybe deleting cookies might help with the token error as

might making use of browser.call( function () {

  1. speed vs robustness: We'd like the tests to be more robust and open a new browser instance after every test. If a test logs in and doesn't log out, all subsequent tests run in login state. Z says we can do this and can do it in MobileFrontend. opening/closing can add 10s but more robustness is a good thing. Alternatively if this speed cost is unacceptable clearing cookies might be enough. There's a setting in wdio-config that can be added to automatically do that. Action: create a task around using new browser sessions.
  1. cucumber - there are plans to support it, but not there yet. We hoped to replicate the Ruby stack, but various developers do not like Cucumber and the cost of abstraction that comes with it. There are benefits of using it (non-technical people can make sense of tests). Primary goal is communication. Outside reading web (which have chores), non-technical people are not looking at browser tests. For majority of teams, Cucumber not necessary. Reading web will need to decide whether they need it or not and are willing to wait.

Some useful links:

In terms of next steps, it looks like we successfully unblocked T221860 by learning about the browser.call function (and the caveat around wrapping any asynchronous code in that method. As we continue work on the epic T174018 we can go at a much slower pace and have discussions as needed.

Reading web must make a decision about whether we want to use Cucumber. I've added a note to myself to find a time to talk about this at the offsite.