Page MenuHomePhabricator

Evaluate Cypress as a WebdriverIO replacement for our browser automation framework
Closed, ResolvedPublic

Related Objects

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Change 530914 abandoned by Zfilipin:
WIP Cypress

Reason:
just a test

https://gerrit.wikimedia.org/r/530914

Change 599937 had a related patch set uploaded (by AlQaholic007; owner: AlQaholic007):
[mediawiki/core@master] WIP Cypress: Implement test for checking Admin login

https://gerrit.wikimedia.org/r/599937

Change 600406 had a related patch set uploaded (by AlQaholic007; owner: AlQaholic007):
[mediawiki/core@master] WIP Cypress: Implement test to check for account creation

https://gerrit.wikimedia.org/r/600406

Change 597125 had a related patch set uploaded (by AlQaholic007; owner: AlQaholic007):
[mediawiki/core@master] WIP Cypress: Install dependencies

https://gerrit.wikimedia.org/r/597125

Change 598253 had a related patch set uploaded (by AlQaholic007; owner: AlQaholic007):
[mediawiki/core@master] WIP Cypress: Implement a simple test to check for default main page heading

https://gerrit.wikimedia.org/r/598253

Change 598253 abandoned by AlQaholic007:
WIP Cypress: Implement a simple test to check for default main page heading

Reason:
Too many conflicts

https://gerrit.wikimedia.org/r/598253

Change 599937 abandoned by AlQaholic007:
WIP Cypress: Implement test for checking Admin login

Reason:
Too many conflicts

https://gerrit.wikimedia.org/r/599937

Change 600406 abandoned by AlQaholic007:
WIP Cypress: Implement test to check for account creation

Reason:
Too many conflicts

https://gerrit.wikimedia.org/r/600406

Change 601343 had a related patch set uploaded (by AlQaholic007; owner: AlQaholic007):
[mediawiki/core@master] WIP Cypress: Implement a simple test to check for main page heading

https://gerrit.wikimedia.org/r/601343

Change 601345 had a related patch set uploaded (by AlQaholic007; owner: AlQaholic007):
[mediawiki/core@master] WIP Cypress: Implement test to check for user account creation

https://gerrit.wikimedia.org/r/601345

Change 601375 had a related patch set uploaded (by AlQaholic007; owner: AlQaholic007):
[mediawiki/core@master] WIP Cypress: Implement test to check for user login @daily

https://gerrit.wikimedia.org/r/601375

Change 601746 had a related patch set uploaded (by AlQaholic007; owner: AlQaholic007):
[mediawiki/core@master] WIP Cypress: Page should be previewable

https://gerrit.wikimedia.org/r/601746

Change 601752 had a related patch set uploaded (by AlQaholic007; owner: AlQaholic007):
[mediawiki/core@master] WIP Cypress: Page should be creatable

https://gerrit.wikimedia.org/r/601752

Change 601752 abandoned by AlQaholic007:
WIP Cypress: Page should be creatable

Reason:
Flaky test and lot of conflicts

https://gerrit.wikimedia.org/r/601752

Change 601845 had a related patch set uploaded (by AlQaholic007; owner: AlQaholic007):
[mediawiki/core@master] WIP Cypress: Page should be creatable

https://gerrit.wikimedia.org/r/601845

Change 601851 had a related patch set uploaded (by AlQaholic007; owner: AlQaholic007):
[mediawiki/core@master] WIP Cypress: Page should be re-creatable

https://gerrit.wikimedia.org/r/601851

Change 598257 had a related patch set uploaded (by AlQaholic007; owner: AlQaholic007):
[mediawiki/core@master] WIP Puppeteer: Install dependencies

https://gerrit.wikimedia.org/r/598257

zeljkofilipin updated the task description. (Show Details)
zeljkofilipin updated the task description. (Show Details)
zeljkofilipin moved this task from In Progress to Waiting/Blocked on the User-Soham board.
zeljkofilipin moved this task from In Progress to Waiting/Blocked on the User-Soham board.

Change 597125 abandoned by AlQaholic007:
[mediawiki/core@master] WIP Cypress: Install dependencies

Reason:

https://gerrit.wikimedia.org/r/597125

Change 601343 abandoned by AlQaholic007:
[mediawiki/core@master] WIP Cypress: A sample test

Reason:

https://gerrit.wikimedia.org/r/601343

Change 601345 abandoned by AlQaholic007:
[mediawiki/core@master] WIP Cypress: User should be able to create account

Reason:

https://gerrit.wikimedia.org/r/601345

Change 601375 abandoned by AlQaholic007:
[mediawiki/core@master] WIP Cypress: User should be able to login

Reason:
GSOC Completion

https://gerrit.wikimedia.org/r/601375

Change 601746 abandoned by AlQaholic007:
[mediawiki/core@master] WIP Cypress: Page should be previewable

Reason:
GSOC Completion

https://gerrit.wikimedia.org/r/601746

Change 601845 abandoned by AlQaholic007:
[mediawiki/core@master] WIP Cypress: Page should be creatable

Reason:
GSOC Completion

https://gerrit.wikimedia.org/r/601845

Change 601851 abandoned by AlQaholic007:
[mediawiki/core@master] WIP Cypress: Page should be re-creatable

Reason:
GSOC Completion

https://gerrit.wikimedia.org/r/601851

Hi @zeljkofilipin @Jpita

I don't see much movement on this task in the past 6 months, and would great to have an update here as we are about to rewrite our old ruby based browser tests and would like to consider the results of this investigation etc in terms of the target that we go for.

See T180228#7178058

Hi @Addshore

I'd written a post on our evaluations of the frameworks. Unfortunately back in the time we had some issues with Cypress that couldn't be resolved at the time. I think there's an existing patch on that.

I'll link the post below:

https://sohamp.dev/blog/2020-08-24-wmf-gsoc2020/

Hope that helps

@Addshore do u have any preference?
Did your team discuss this?

I'd written a post on our evaluations of the frameworks. Unfortunately back in the time we had some issues with Cypress that couldn't be resolved at the time. I think there's an existing patch on that.

I'll link the post below:

https://sohamp.dev/blog/2020-08-24-wmf-gsoc2020/

Thanks for the reply.
We include a link to your blog post over in the puppeteer section of T180228#7178058 actually!
thanks for the content there

@Michael took a look at that patch and seemed to be able to get it working.

@Addshore do u have any preference?
Did your team discuss this?

We have had various discussion in the past, not are probably just about to set a precedent with the rewriting of these ruby browser tests for our projects.
We are probably mainly looking between Cypress and Puppeteer at this stage, but it is certainly still an open discussion / discussion that is about to happen.
We already have one owned code base that uses Cypress and runs its CI etc on Jenkins.

What we would want to avoid is that we make a decision, do a bunch of work, and in one months time this task reveals that potentially someone had strong feelings in the other direction!

Next steps
  • We need to decide if we want to spend some time evaluating the alternatives
    • We should, because of the reasons mentioned above
  • We should write a short ADR on either why we want to continue using WebdriverIO + Selenium, or why decided to go with one of the alternatives

I'm of the opinion that each team should use whatever tool they see that fits better.

At the inuka team we evaluated Selenium and didn't like it so we went with Cypress

https://github.com/wikimedia/wikipedia-kaios/tree/main/cypress

zeljkofilipin changed the task status from Open to In Progress.Sep 16 2021, 10:28 AM
zeljkofilipin claimed this task.

Evaluation finished.

Can I ask where the evaluation results landed? I see that the blog post mentions some trouble when getting Cypress integrated into Wikimedia CI, was there any other activity we can learn from?

My context is that my team has tried Cypress locally and it seems good--better than selenium, we have a docker command which can run the tests headless, and I see a small amount of cypress peppered around Wikimedia repositories, mostly in an experimental way.

[...] was there any other activity we can learn from?

Yes, we're increasingly using cypress and as far as I can tell, it mostly just works out of the box. Just plug it in to the selenium-test command in your package.json's scripts section.

One example where we're using it in a MediaWiki extension would be EntitySchema: https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/extensions/EntitySchema/+/refs/heads/master/package.json#13 (If I reread the conversations correctly (see Add Cypress (Ifd127298)), then the cache-env variable is for running it locally, and you already got that solved)

@awight various selenium/webdriverio alternatives were evaluated in several internships. As far as I remember, puppeteer, cypress and playwright. All of them worked just fine. Some of them were faster, some had some nice features that are not present in webdriverio. The reason we didn't switch to any of them is that we have existing codebase in webdriverio. None of the alternatives were so better that the investment in rewriting all existing tests would be a good investment.

We are already pretty busy trying to keep webdriverio tests up to date. For example, by updating all repositories to the latest major version of webdriverio. We do not have the expertise to do that for any other framework.

It has been a few years since we did the evaluations. I forgot that we had some trouble with cypress in CI early on. As far as I remember, cypress worked just fine later.

Some teams have decided to rewrite webdriverio tests to cypress. Our CI should support that. (Citation needed.)

I'm currently traveling so I have limited time for phabricator archeology, but I can find other evaluations. I'm also happy to talk more about this in slack or a meeting, what ever makes sense.

Thank you, that's already very helpful information! I ask because my team (WMDE Tech Wishes) has been experimenting with cypress and the experience is good so far. We had the advantage of starting with Michael's integration work, so there was no heavy lifting involved.

I guess the question at this point is how the test and infrastructure engineers feel about a proliferation of end-to-end testing frameworks. For the moment we're fine with not getting official support, but I think the problematic lifecycle phase begins once we've moved on to the next project and our work is sometimes maintained by others.