Brief Summary:
The Mediawiki Cite extension currently uses the Cypress framework for the end-to-end tests. This project will take an existing Cypress test, implement the same one with the Selenium framework, and run both tests 100 times in the CI.
Thereby, we want to compare the 2 frameworks on factors such as: execution speed, flakiness, concurrency. The Cypress tests are currently not set up for concurrency. We’ll try looking into the feasibility of that and will document our insights.
Expected outcomes:
- Set up a metric (Run tests 100 times in the CI), and then report on run time and flakiness rate.
- Try to get Cypress to run in parallel on the CI
- Give pros and cons of both frameworks
Skills preferred:
Basic JavaScript, CI
Experience Level:
Anyone is welcome, even devs that have never written an automated test before
Project Leads:
@WMDE-Fisch , @MareikeHeuerWMDE
Starting tasks:
to be created
Why are you proposing this project?
We migrated from Selenium to Cypress for the Mediawiki Cite extension. As other teams might consider switching, we want to compare the performance and provide a comparative assessment. At the end, we will have real numbers, as well as an assessment on concurrency difficulty, which would help Wikimedians make their choice.
What is the expected impact?
- Helping others make an informed decision when choosing their framework
- Report on if the switch to Cypress was beneficial despite concurrency limits