Prio Notes:

When pulled into board, let's discuss what timebox should be given for this

Change #1094415 had a related patch set uploaded (by Arthur taylor; author: Arthur taylor):

[mediawiki/extensions/WikibaseLexeme@master] Use `cypress-cloud` to support multiple parallel agents for cypress

https://gerrit.wikimedia.org/r/1094415

Investigated the option of using cypress-cloud / currents.dev / sorry-cypress. Sorry Cypress provides an open-source, self-hosted equivalent of the cloud / SaaS cypress.io. A local version of the server-side can be run with:

$ docker run -p 1234:1234 agoldis/sorry-cypress-director

and starts listening for connections on https://localhost:1234. Combined with the attached patch, this in principle allows cypress test suite executions to be scaled across multiple agents / CI runners.

That's all well and good, but unfortunately it seems that Sorry-Cypress / currents.dev picked a fight with Cypress and lost. Cypress doesn't care about abandoning its users and cypress-cloud / currents only provides limited support on their beta channel - they appear to have pivoted to Playwright.

This seems to be a great example of greedy companies and the failures of the open-core funding model. It's bad news for our efforts to scale up cypress testing, and for me it calls into question the choice of using cypress at all.

timebox: 8 hours

Change #1097353 had a related patch set uploaded (by Arthur taylor; author: Arthur taylor):

[integration/config@master] Add experimental agents for distributed parallel cypress runs

https://gerrit.wikimedia.org/r/1097353

Change #1097356 had a related patch set uploaded (by Arthur taylor; author: Arthur taylor):

[mediawiki/extensions/WikibaseLexeme@master] Use `@badeball/cypress-parallel` for parallel runs

https://gerrit.wikimedia.org/r/1097356

Added two more patches (1, 2) looking at a different approach for multi-agent parallelism. @badeball/cypress-parallel takes the approach of splitting the tests locally and relying on an environment cue to determine which block of tests to run. In my local testing, I get the following for the cypress tests in WikibaseLexeme:

$ SELENIUM_PARALLEL_NODE_ID=1 MEDIAWIKI_USER=root MEDIAWIKI_PASSWORD=[password] MW_SERVER=http://wikidatawikidev.mediawiki.mwdd.localhost:8080/ MW_SCRIPT_PATH=w/ npm run cypress:parallel

Node 1: 01:10
Node 2: 00:56
Node 3: 01:16
Node 4: 00:55

vs. linear:

$ MEDIAWIKI_USER=root MEDIAWIKI_PASSWORD=[password] MW_SERVER=http://wikidatawikidev.mediawiki.mwdd.localhost:8080/ MW_SCRIPT_PATH=w/ npm run cypress:run

03:32

The second patch is a suggestion for how that might look in CI, but it is not tested and the approach has not been reviewed by WMF.

The @badeball/cypress-parallel library is not super active or popular. There are a number of forks and other packages with similar names and it's not clear how well-maintained the library is. But it is also a small library with limited functionality - it seems within our capacities to adopt the project if it gets abandoned by its maintainer, and it doesn't seem in direct competition with Cypress' business model (so it's unlikely to get sabotaged by the Cypress team).

Also worth noting here that wdio supports sharding out of the box . An equivalent agent setup to that outlined above could work for both frameworks, and would be transparent to the Quibble / Jenkins side.

FWIW, if we decide to go with parallelizing the browser tests inside one “test runner”, going with @badeball/cypress-parallel sounds fine to me – it’s not ideal, but if it breaks the worst case is probably just that we have to take it out and suffer longer build times again. But the general approach is also under discussion in T374003 at the moment.