Run browser tests in parallel
Open, Needs TriagePublic
Actions

Assigned To

None

Authored By

	awight
	Jun 28 2019, 8:58 PM

Description

(Apologies if this is a duplicate, I know there's been some discussion already but couldn't find it in Phabricator.)

Browser tests are slow, and there are a few opportunities for parallelization. The easiest step would be to run our QUnit and node-selenium tests in separate threads. After T199116 we could also run selenium tests for each repo in parallel, but this introduces the additional challenge of potential interaction between tests.

However, there are some questions to work through first:

Can our test webserver handle multiple clients? (See also T225218.)
Is it reasonable to run multiple chromedrivers on one machine? How much more memory will this require?
How many of our tests will become fragile, for example because they depend on a constant rather than randomly-generated title?

Details

Related Changes in Gerrit:

Subject	Repo	Branch	Lines +/-
selenium: run tests concurrently	mediawiki/extensions/Wikibase	master	+2 -0
selenium: run tests concurrently	mediawiki/extensions/ProofreadPage	master	+2 -1
selenium: run tests concurrently	mediawiki/extensions/CampaignEvents	master	+3 -2
selenium: run tests concurrently	mediawiki/extensions/GrowthExperiments	master	+1 -0
selenium: run tests concurrently	mediawiki/extensions/Wikibase	master	+3 -3
selenium: run tests concurrently	mediawiki/extensions/PageTriage	master	+1 -0
[WIP] Run Selenium tests for each project in parallel	integration/quibble	master	+46 -72
selenium: Run tests in each suite concurrently	mediawiki/extensions/WikibaseLexeme	master	+2 -0
BrowserTests: Support splitting projects into two groups	integration/quibble	master	+17 -0
selenium: run tests concurrently	mediawiki/extensions/CheckUser	master	+96 -9
selenium: run tests concurrently	mediawiki/extensions/VisualEditor	master	+3 -1
selenium: run tests concurrently	mediawiki/extensions/Echo	master	+3 -1
selenium: run tests concurrently	mediawiki/extensions/CheckUser	master	+3 -1
selenium: Run test suites concurrently	mediawiki/extensions/GrowthExperiments	master	+1 -0
Release Quibble 1.3.0	integration/quibble	master	+38 -3
selenium: Run test suites concurrently	mediawiki/core	master	+1 -1
selenium: Run test suites concurrently	mediawiki/extensions/AbuseFilter	master	+2 -1
selenium: run tests concurrently	mediawiki/skins/MinervaNeue	master	+3 -1
selenium: run 4 tests in parallel	mediawiki/core	master	+2 -0
BrowserTests: Option to run npm install in parallel	integration/quibble	master	+55 -7
Prepare to run browser suites in parallel	integration/quibble	master	+145 -64

Related Objects
Search...

Status	Subtype	Assigned	Task
Open		None	T76586 K3. Run Flow tests with Parsoid on Jenkins
Resolved	Feature	kostajh	T218534 Run Parsoid service in quibble
Open		None	T225730 Reduce runtime of MW shared gate Jenkins jobs to 5 min
Open		None	T226869 Run browser tests in parallel
Declined		None	T291476 Migrate Wikibase selenium tests to Quibble+Apache, enable concurrency
Resolved		None	T285649 Switch all Quibble Selenium and api-testing jobs to use apache
Resolved		None	T276428 Introduce non-voting jobs with quibble+apache
Resolved		awight	T225218 Consider httpd for quibble instead of php built-in server
Resolved		kostajh	T282154 Wikibase: Special:EntityData compression encoding issues in local dev env
Open		None	T227352 Set up extension tests for Parsoid repo
Resolved		cscott	T248726 Run phan on code in extension/
Resolved		kostajh	T297480 Inject environment variable to indicate that Quibble is running with Apache
Resolved		kostajh	T298514 "should mark the revert as a bot edit" test flaky with Apache + php-fpm
Resolved		kostajh	T299491 Switch QUnit tests to use Apache backend
Resolved		hashar	T299492 Quibble ci-fullrun jobs should use Apache backend
Open		None	T301100 Fix flaky ContributionsCount.js test
Resolved	PRODUCTION ERROR	hoo	T255566 [LEX] [TECH] Figure out why LexemeEditEntityTest::testGivenInvalidDataInClearRequest_errorIsReported is throwing an Exception/logging an error with AbuseFilter enabled
Resolved		Lucas_Werkmeister_WMDE	T298682 Wikibase\Repo\Store\Sql\SqlIdGenerator::generateNewId deadlock on wb_id_counters when running selenium tests in parallel
Resolved		None	T337862 Try to run browser tests of WMDE-TechWish projects in parallel
Resolved		Dreamy_Jazz	T340823 Fix CheckUser selenium tests so they can run concurrently
Resolved		Krinkle	T199393 Selenium tests sometimes fail due to deadlock in User::addToDatabase from api.php?action=createaccount
Resolved		hashar	T342088 Run Quibble Mariadb server with innodb_print_all_deadlocks
Resolved		None	T340943 TwoColConflict selenium tests run in parallel trigger db errors
Open		Peter	T344754 Browser tests video capture is shared between tests
Open		None	T345149 Special:RecentChanges Selenium test has a race condition when running concurrently
Resolved	Spike	zeljkofilipin	T402397 Increase maxInstances for WebdriverIO test in core
Open	Spike	zeljkofilipin	T402927 Investigate why increasing maxInstances for WebdriverIO causes tests to fail

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Change 751940 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[mediawiki/extensions/AbuseFilter@master] selenium: Run tests in each suite concurrently

https://gerrit.wikimedia.org/r/751940

Change 751942 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[mediawiki/extensions/WikibaseLexeme@master] selenium: Run tests in each suite concurrently

https://gerrit.wikimedia.org/r/751942

Change 751957 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[mediawiki/extensions/GrowthExperiments@master] selenium: Run test suites concurrently

https://gerrit.wikimedia.org/r/751957

Change 751959 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[mediawiki/core@master] selenium: Run test suites concurrently

https://gerrit.wikimedia.org/r/751959

Change 751961 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[mediawiki/core@master] api-testing: Run tests in parallel

https://gerrit.wikimedia.org/r/751961

Change 751940 merged by jenkins-bot:

[mediawiki/extensions/AbuseFilter@master] selenium: Run test suites concurrently

https://gerrit.wikimedia.org/r/751940

kostajh mentioned this in T298735: Run api-testing tests in parallel.Jan 6 2022, 9:19 PM

Change 751959 abandoned by Kosta Harlan:

[mediawiki/core@master] selenium: Run test suites concurrently

Reason:

https://gerrit.wikimedia.org/r/751959

zeljkofilipin moved this task from Watching 📺 to Deep work 🌊 on the User-zeljkofilipin board.Jan 7 2022, 1:29 PM

zeljkofilipin moved this task from Deep work 🌊 to Watching 📺 on the User-zeljkofilipin board.Jan 12 2022, 3:36 PM

Umherirrender mentioned this in T299077: CI job wmf-quibble-selenium-php72-docker get a deadlock on `change_tag_def` (NameTableStore).Jan 12 2022, 5:58 PM

Change 754524 had a related patch set uploaded (by Hashar; author: Hashar):

[integration/quibble@master] Release Quibble 1.3.0

https://gerrit.wikimedia.org/r/754524

Change 754524 merged by jenkins-bot:

[integration/quibble@master] Release Quibble 1.3.0

https://gerrit.wikimedia.org/r/754524

Mentioned in SAL (#wikimedia-releng) [2022-01-17T16:16:21Z] <hashar> Tagged Quibble 1.3.0 @ 2b2c7f9a45 # T297480 T226869 T294931

Lucas_Werkmeister_WMDE closed subtask T298682: Wikibase\Repo\Store\Sql\SqlIdGenerator::generateNewId deadlock on wb_id_counters when running selenium tests in parallel as Resolved.Jan 18 2022, 3:27 PM

thiemowmde added a project: WMDE-TechWish-Maintenance.Jan 31 2022, 10:08 AM

Change 768354 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[integration/quibble@master] [WIP] Run Selenium tests for each project in parallel

https://gerrit.wikimedia.org/r/768354

In T226869#7755609, @gerritbot wrote:

Change 768354 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[integration/quibble@master] [WIP] Run Selenium tests for each project in parallel

https://gerrit.wikimedia.org/r/768354

This is not a million miles off. Some observations:

The Apache config needs tuning, as the "Server unavailable" error shows up a couple of times:

Some tests fail on first attempt due to interference of other tests (or due to the 503s noted above), but then they succeed due to the automatic retries in wdio. Not ideal, but it mostly seems to work out.
Some tests need to be reworked slightly to be more robust. E.g. one of the Echo tests assumes the account isn't logged-in, but it is.
Some tests simulate blocking a user and then other tests which use the same user fail, because the user is blocked (https://integration.wikimedia.org/ci/job/integration-quibble-fullrun-extensions/75/artifact/log/Lexeme%253AForms-can-edit-statements-on-a-new-Form-2022-03-06T21-57-01-255Z.mp4)

@zeljkofilipin I was wondering if we could consider using a single wdio.conf.js in mediawiki core that each extension references. That way, we'd get WDIO's parallelization for free, as it already supports concurrent test running, if it knows about all the tests that need to be run.

Core's wdio.conf.js file would need to be adjusted to look in extensions/skins:

	specs: [
		'./tests/selenium/specs/**/*.js',
                './extensions/**/tests/selenium/specs/**/*.js'
	],

and each extension/skin would have to migrate its wdio.conf.js contents into before/beforeEach methods in specFiles, rather than in wdio.conf.js

Having a single configuration file for MW core + extensions/skins seems more akin to how we do things for PHPUnit.

thiemowmde moved this task from Incoming to Watching or blocked on the WMDE-TechWish-Maintenance board.May 5 2022, 11:13 AM

zeljkofilipin moved this task from Watching 📺 to Deep work 🌊 on the User-zeljkofilipin board.May 6 2022, 12:34 PM

In T226869#7899039, @kostajh wrote:
@zeljkofilipin I was wondering if we could consider using a single wdio.conf.js in mediawiki core that each extension references. That way, we'd get WDIO's parallelization for free, as it already supports concurrent test running, if it knows about all the tests that need to be run.

Core's wdio.conf.js file would need to be adjusted to look in extensions/skins:
	specs: [
		'./tests/selenium/specs/**/*.js',
                './extensions/**/tests/selenium/specs/**/*.js'
	],
and each extension/skin would have to migrate its wdio.conf.js contents into before/beforeEach methods in specFiles, rather than in wdio.conf.js

Having a single configuration file for MW core + extensions/skins seems more akin to how we do things for PHPUnit.

Note that if we do this, some Quibble changes would be necessary (cc @hashar) because Quibble currently assumes a wdio specific config for each extension/skin that contains a list of tests to run for just that extension/skin. And that's how Quibble knows, for example, to first run tests for the extension/skin under test. Although, if we are running everything together in parallel, I guess it's not so important anymore to run the extension/skin browser tests ahead of the other ones.

Although PHPUnit integration tests and QUnit are centrally set in mediawiki/core, webdriver.io tests are split in each repositories. That is the model also used for to the linters (which are run via composer test and npm test) and let us use different versions of webdriver.io. It is too challenging if not impossible to force migrate all repositories at the same time.

From the list of attached patches there are a few repositories for which tests are broken when run concurrently (see patches associated to this task: bug:T226869 is:open. I imagine if we ran tests from different extensions in parallel we would encounter even more issues.

Lens0021 unsubscribed.May 12 2022, 4:03 PM

zeljkofilipin moved this task from Deep work 🌊 to Quarter 👔 on the User-zeljkofilipin board.May 13 2022, 12:40 PM

In T226869#7899039, @kostajh wrote:

@zeljkofilipin I was wondering if we could consider using a single wdio.conf.js in mediawiki core that each extension references.

Sorry, looks like I didn't reply to your question. 🤦‍♂️

As far as I remember, that's how we started with the webdriver tests. All configuration and dependencies were in core. That simplified some things (like having to update configuration in just one place, core) but vastly complicated some other things (like making a breaking change in core causing a lot of tests in a lot of repositories to fail). I forgot all the details, but I think the consensus was that having to do more work (like updating configuration/dependencies in each repository) was much better long-term than having everything centralized in core.

@hashar and @Krinkle might remember more details.

zeljkofilipin moved this task from Quarter 👔 to Watching 📺 on the User-zeljkofilipin board.Jun 24 2022, 12:49 PM

Change 751957 abandoned by Kosta Harlan:

[mediawiki/extensions/GrowthExperiments@master] selenium: Run test suites concurrently

Reason:

This is not going to be easy to implement, and not worthwhile at the moment given the relatively limited number of tests.

https://gerrit.wikimedia.org/r/751957

Michael mentioned this in T331732: [ES-M1] Investigate browser testing frameworks for Entity Schema Extension.Mar 17 2023, 1:08 PM

Change 934329 had a related patch set uploaded (by WMDE-Fisch; author: WMDE-Fisch):

[mediawiki/extensions/VisualEditor@master] selenium: run tests concurrently

https://gerrit.wikimedia.org/r/934329

Change 934331 had a related patch set uploaded (by WMDE-Fisch; author: WMDE-Fisch):

[mediawiki/extensions/Echo@master] selenium: run tests concurrently

https://gerrit.wikimedia.org/r/934331

Change 934333 had a related patch set uploaded (by WMDE-Fisch; author: WMDE-Fisch):

[mediawiki/extensions/CheckUser@master] selenium: run tests concurrently

https://gerrit.wikimedia.org/r/934333

Change 934333 merged by jenkins-bot:

[mediawiki/extensions/CheckUser@master] selenium: run tests concurrently

https://gerrit.wikimedia.org/r/934333

Change 934331 merged by jenkins-bot:

[mediawiki/extensions/Echo@master] selenium: run tests concurrently

https://gerrit.wikimedia.org/r/934331

Change 934329 merged by jenkins-bot:

[mediawiki/extensions/VisualEditor@master] selenium: run tests concurrently

https://gerrit.wikimedia.org/r/934329

ReleaseTaggerBot added a project: MW-1.41-notes (1.41.0-wmf.16; 2023-07-04).Jun 30 2023, 1:01 PM

Change 935162 had a related patch set uploaded (by Dreamy Jazz; author: WMDE-Fisch):

[mediawiki/extensions/CheckUser@master] selenium: run tests concurrently

https://gerrit.wikimedia.org/r/935162

WMDE-Fisch closed subtask T337862: Try to run browser tests of WMDE-TechWish projects in parallel as Resolved.Jul 18 2023, 8:10 AM

Daimona mentioned this in T225730: Reduce runtime of MW shared gate Jenkins jobs to 5 min.Jul 18 2023, 1:00 PM

When crafting a small change to Selenium test https://gerrit.wikimedia.org/r/c/mediawiki/core/+/842391 the specs/recentchanges.js fail. It grabs the list of titles from recent change, pick the first title and fail cause another test created some other page so the expected title is no more the first in the entry. When looking at the captured output, I see the browser window creating pages with the title prefix BeforeEach-name-, however that does not come from spec/recentchanges.js but from specs/page.js!

In MediaWiki core I have set maxInstances: 4, what I suspect is that all four browser share the same X display and the video capture for a given test ends up having frame from another test running concurrently! I have no idea how to fix that though :)

Edit: I have filed the above as T344754: Browser tests video capture is shared between tests

hashar mentioned this in T344754: Browser tests video capture is shared between tests.Aug 22 2023, 6:05 PM

hashar added a subtask: T344754: Browser tests video capture is shared between tests.

hashar added a subtask: T345149: Special:RecentChanges Selenium test has a race condition when running concurrently.Aug 29 2023, 12:49 PM

Change 935162 merged by jenkins-bot:

[mediawiki/extensions/CheckUser@master] selenium: run tests concurrently

https://gerrit.wikimedia.org/r/935162

Dreamy_Jazz closed subtask T340823: Fix CheckUser selenium tests so they can run concurrently as Resolved.Aug 31 2023, 2:31 PM

Change 747904 abandoned by Hashar:

[integration/quibble@master] BrowserTests: Support splitting projects into two groups

Reason:

https://gerrit.wikimedia.org/r/747904