We have recently done a bit of refactoring to the browser tests for cirrussearch to get the time they take to run under control. Using parallel_cucumber with 4 processes they are down to around 5 minutes (or more depending on load of the server tests are running against).
This should initially be non-voting, but will be made to vote once we see that it doesn't have intermittent failures and such.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | zeljkofilipin | T91669 Make browser tests voting for all repos of WMF deployed code | |||
| Resolved | zeljkofilipin | T55697 [EPIC] trigger browser tests from Gerrit (tracking) | |||
| Declined | None | T98374 Make browsertests for CirrusSearch run on every submitted patch with proper CI infrastructure rather than a bot |
I should note that while they are called browsertests since they are in cucumber, the majority of the tests now talk directly to the api and do not use the browser. If it makes things easier we can run only the @api tag and skip saucelabs,
Running just the api tests (bundle exec parallel_cucumber features/ -o '--tags @api) in four processes against a 4 core labs instance does 394 scenarios in ~4 minutes.
Running the entire suite (bundle exec parallel_cucumber features/) in four processes against a 4 core labs instance does 502 scenarios in ~5.5 minutes
@dduvall added support for that to the CI infrastructure (T103039).
Reading has an epic about it T104315: [GOAL]: Set up testing environments that run browser tests on all patches. and they now have browser tests voting for some repositories ( T100293: [EPIC] Run browser tests against every MobileFrontend/Gather patch ).
@EBernhardson can you sync up with @Jdlrobson, @Jhernandez and @dduvall ?
This was resolved with the creation of Cindy: https://gerrit.wikimedia.org/r/#/q/reviewer:%22Cindy-the-browser-test-bot+%253Cbernhardsonerik%2540gmail.com%253E%22,n,z
Then I'll have to take it out of Discovery, because it's pretty far down our list of things to fix.
Currently Cindy has special setup code for these test, that will need to be adapted (will be changed to not break other tests in T120360).
Unlikely to ever happen because of T139740: Port Selenium tests from Ruby to Node.js. Please reopen if you would like to make it happen.