Documentation:
Examples:
Micro-tasks:
Mentors:
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | zeljkofilipin | T247833 Mentor a Google Summer of Code 2020 student | |||
| Resolved | zeljkofilipin | T247835 Evaluate WebdriverIO replacements for our browser automation framework | |||
| Resolved | zeljkofilipin | T230729 Evaluate Cypress as a WebdriverIO replacement for our browser automation framework | |||
| Resolved | Jdforrester-WMF | T247599 install cypress dependencies on CI | |||
| Resolved | zeljkofilipin | T248220 Micro-task: Create a simple test using Cypress | |||
| Duplicate | None | T249443 Cypress as a WebDriverIO replacement of Mediawiki's browser automation | |||
| Declined | None | T249450 Getting Login Session Error when running tests with Cypress | |||
| Declined | None | T254775 Cypress: "Page should be creatable" test fails on the CI because of welcomedialog pop up on TestWiki | |||
| Declined | None | T256389 Evaluate Cypress in mediawiki/core | |||
| Declined | None | T256390 Evaluate Cypress in EntitySchema |
Change 599937 had a related patch set uploaded (by AlQaholic007; owner: AlQaholic007):
[mediawiki/core@master] WIP Cypress: Implement test for checking Admin login
Change 600406 had a related patch set uploaded (by AlQaholic007; owner: AlQaholic007):
[mediawiki/core@master] WIP Cypress: Implement test to check for account creation
Change 597125 had a related patch set uploaded (by AlQaholic007; owner: AlQaholic007):
[mediawiki/core@master] WIP Cypress: Install dependencies
Change 598253 had a related patch set uploaded (by AlQaholic007; owner: AlQaholic007):
[mediawiki/core@master] WIP Cypress: Implement a simple test to check for default main page heading
Change 598253 abandoned by AlQaholic007:
WIP Cypress: Implement a simple test to check for default main page heading
Reason:
Too many conflicts
Change 599937 abandoned by AlQaholic007:
WIP Cypress: Implement test for checking Admin login
Reason:
Too many conflicts
Change 600406 abandoned by AlQaholic007:
WIP Cypress: Implement test to check for account creation
Reason:
Too many conflicts
Change 601343 had a related patch set uploaded (by AlQaholic007; owner: AlQaholic007):
[mediawiki/core@master] WIP Cypress: Implement a simple test to check for main page heading
Change 601345 had a related patch set uploaded (by AlQaholic007; owner: AlQaholic007):
[mediawiki/core@master] WIP Cypress: Implement test to check for user account creation
Change 601375 had a related patch set uploaded (by AlQaholic007; owner: AlQaholic007):
[mediawiki/core@master] WIP Cypress: Implement test to check for user login @daily
Change 601746 had a related patch set uploaded (by AlQaholic007; owner: AlQaholic007):
[mediawiki/core@master] WIP Cypress: Page should be previewable
Change 601752 had a related patch set uploaded (by AlQaholic007; owner: AlQaholic007):
[mediawiki/core@master] WIP Cypress: Page should be creatable
Change 601752 abandoned by AlQaholic007:
WIP Cypress: Page should be creatable
Reason:
Flaky test and lot of conflicts
Change 601845 had a related patch set uploaded (by AlQaholic007; owner: AlQaholic007):
[mediawiki/core@master] WIP Cypress: Page should be creatable
Change 601851 had a related patch set uploaded (by AlQaholic007; owner: AlQaholic007):
[mediawiki/core@master] WIP Cypress: Page should be re-creatable
Change 598257 had a related patch set uploaded (by AlQaholic007; owner: AlQaholic007):
[mediawiki/core@master] WIP Puppeteer: Install dependencies
Change 597125 abandoned by AlQaholic007:
[mediawiki/core@master] WIP Cypress: Install dependencies
Reason:
Change 601343 abandoned by AlQaholic007:
[mediawiki/core@master] WIP Cypress: A sample test
Reason:
Change 601345 abandoned by AlQaholic007:
[mediawiki/core@master] WIP Cypress: User should be able to create account
Reason:
Change 601375 abandoned by AlQaholic007:
[mediawiki/core@master] WIP Cypress: User should be able to login
Reason:
GSOC Completion
Change 601746 abandoned by AlQaholic007:
[mediawiki/core@master] WIP Cypress: Page should be previewable
Reason:
GSOC Completion
Change 601845 abandoned by AlQaholic007:
[mediawiki/core@master] WIP Cypress: Page should be creatable
Reason:
GSOC Completion
Change 601851 abandoned by AlQaholic007:
[mediawiki/core@master] WIP Cypress: Page should be re-creatable
Reason:
GSOC Completion
I don't see much movement on this task in the past 6 months, and would great to have an update here as we are about to rewrite our old ruby based browser tests and would like to consider the results of this investigation etc in terms of the target that we go for.
See T180228#7178058
Hi @Addshore
I'd written a post on our evaluations of the frameworks. Unfortunately back in the time we had some issues with Cypress that couldn't be resolved at the time. I think there's an existing patch on that.
I'll link the post below:
https://sohamp.dev/blog/2020-08-24-wmf-gsoc2020/
Hope that helps
Thanks for the reply.
We include a link to your blog post over in the puppeteer section of T180228#7178058 actually!
thanks for the content there
@Michael took a look at that patch and seemed to be able to get it working.
We have had various discussion in the past, not are probably just about to set a precedent with the rewriting of these ruby browser tests for our projects.
We are probably mainly looking between Cypress and Puppeteer at this stage, but it is certainly still an open discussion / discussion that is about to happen.
We already have one owned code base that uses Cypress and runs its CI etc on Jenkins.
What we would want to avoid is that we make a decision, do a bunch of work, and in one months time this task reveals that potentially someone had strong feelings in the other direction!
I'm of the opinion that each team should use whatever tool they see that fits better.
At the inuka team we evaluated Selenium and didn't like it so we went with Cypress
https://github.com/wikimedia/wikipedia-kaios/tree/main/cypress
Can I ask where the evaluation results landed? I see that the blog post mentions some trouble when getting Cypress integrated into Wikimedia CI, was there any other activity we can learn from?
My context is that my team has tried Cypress locally and it seems good--better than selenium, we have a docker command which can run the tests headless, and I see a small amount of cypress peppered around Wikimedia repositories, mostly in an experimental way.
Yes, we're increasingly using cypress and as far as I can tell, it mostly just works out of the box. Just plug it in to the selenium-test command in your package.json's scripts section.
One example where we're using it in a MediaWiki extension would be EntitySchema: https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/extensions/EntitySchema/+/refs/heads/master/package.json#13 (If I reread the conversations correctly (see Add Cypress (Ifd127298)), then the cache-env variable is for running it locally, and you already got that solved)
@awight various selenium/webdriverio alternatives were evaluated in several internships. As far as I remember, puppeteer, cypress and playwright. All of them worked just fine. Some of them were faster, some had some nice features that are not present in webdriverio. The reason we didn't switch to any of them is that we have existing codebase in webdriverio. None of the alternatives were so better that the investment in rewriting all existing tests would be a good investment.
We are already pretty busy trying to keep webdriverio tests up to date. For example, by updating all repositories to the latest major version of webdriverio. We do not have the expertise to do that for any other framework.
It has been a few years since we did the evaluations. I forgot that we had some trouble with cypress in CI early on. As far as I remember, cypress worked just fine later.
Some teams have decided to rewrite webdriverio tests to cypress. Our CI should support that. (Citation needed.)
I'm currently traveling so I have limited time for phabricator archeology, but I can find other evaluations. I'm also happy to talk more about this in slack or a meeting, what ever makes sense.
Thank you, that's already very helpful information! I ask because my team (WMDE Tech Wishes) has been experimenting with cypress and the experience is good so far. We had the advantage of starting with Michael's integration work, so there was no heavy lifting involved.
I guess the question at this point is how the test and infrastructure engineers feel about a proliferation of end-to-end testing frameworks. For the moment we're fine with not getting official support, but I think the problematic lifecycle phase begins once we've moved on to the next project and our work is sometimes maintained by others.