Page MenuHomePhabricator

Run browser tests in parallel
Open, Needs TriagePublic

Description

(Apologies if this is a duplicate, I know there's been some discussion already but couldn't find it in Phabricator.)

Browser tests are slow, and there are a few opportunities for parallelization. The easiest step would be to run our QUnit and node-selenium tests in separate threads. After T199116 we could also run selenium tests for each repo in parallel, but this introduces the additional challenge of potential interaction between tests.

However, there are some questions to work through first:

  • Can our test webserver handle multiple clients? (See also T225218.)
  • Is it reasonable to run multiple chromedrivers on one machine? How much more memory will this require?
  • How many of our tests will become fragile, for example because they depend on a constant rather than randomly-generated title?

Event Timeline

awight created this task.Jun 28 2019, 8:58 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 28 2019, 8:58 PM

Is it reasonable to run multiple chromedrivers on one machine? How much more memory will this require?

I didn't test this, but I'm pretty sure we could reuse one chromedriver for all tests.

How many of our tests will become fragile, for example because they depend on a constant rather than randomly-generated title?

Every test should make sure it's dependencies (users, pages...) are randomly generated and not reused.

WebdriverIO comes with built-in support for running spec files in parallel, up to a configured concurrency. This is tuned with the wdio.conf.js maxInstances setting, which we currently have set to 1. Considering all the things that can go wrong when introducing concurrency here, we should expose the tuning knob to allow for experimentation. For example, we check an environment variable which can be poked through from the Jenkins job.

Meanwhile, I'll experiment with concurrent browser tests in a few repos, to see what obstacles we might run up against.

Change 545650 had a related patch set uploaded (by Awight; owner: Awight):
[mediawiki/extensions/Wikibase@master] [DNM] Experiment with concurrent browser tests

https://gerrit.wikimedia.org/r/545650

Change 545653 had a related patch set uploaded (by Awight; owner: Awight):
[mediawiki/core@master] [DNM] Experiment with concurrent browser tests

https://gerrit.wikimedia.org/r/545653

Change 545650 had a related patch set uploaded (by Awight; owner: Awight):
[mediawiki/extensions/Wikibase@master] [DNM] Experiment with concurrent browser tests

https://gerrit.wikimedia.org/r/545650

This came surprisingly close, only a handful of test cases failed and we might be able to code around the fragility, for example:

22:11:22 1) WikibaseReferenceOnProtectedPage can expand collapsed references on a protected page as unprivileged user:
22:11:22 cantedit: You cannot change the protection levels of this page because you do not have permission to edit it.

... probably due to test cases presuming that side-effects from previous tests will have taken place.

There were no signs of memory exhaustion.

The equivalent patch for mediawiki-core passes :-)

The bigger gain will come when parallelizing at the next level up, however: running suites in parallel.

We should be able to use a single backend DevWebServer, and a single Chrome driver. But since we've isolated each repo's tests in order to decouple dependencies, we can't use WebdriverIO's built-in concurrency. I'll play with Quibble parallelism, executing some configurable maximum number of suites at once.

Change 545661 had a related patch set uploaded (by Awight; owner: Awight):
[integration/quibble@master] [WIP] Run browser suites in parallel

https://gerrit.wikimedia.org/r/545661

Change 545661 abandoned by Awight:
Prepare to run browser suites in parallel

Reason:
Please see Ib2dc728980ce95 instead.

https://gerrit.wikimedia.org/r/545661