instance to easier find flakey tests
Closed, ResolvedPublic5 Estimated Story PointsSpike
Actions

Assigned To

Authored By

	Peter
	Oct 17 2025, 6:57 AM

Description

Let's try to run webdriver.io test and mediawiki (or one of those) on a slow machine/instance. The idea is that if we can slow down the CPU, making tests run slower, tests that are flakey/have timing issues will start to fail. It could be a way for us to stress test our tests for flakiness.

Let's spend a day to try it out. It could be as simple as setting a limit CPU limit on mediawiki-quickstart. Or maybe a dedicated machine.

Do not spend more than 8 hours on the test as first step.

AC:

Try out different ideas and document the result and what you do in this task
If this seems to be a useful, create a follow up task to implement it in our work flow

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Open		None	T407947 Improve visibility and detection of flakey WebdriverIO tests
		Resolved	Spike	pwangai	T407598 Investigate how to run webdriverio/mediawiki on a slow machine/instance to easier find flakey tests

Event Timeline

Peter created this task.Oct 17 2025, 6:57 AM

Restricted Application changed the subtype of this task from "Task" to "Spike". · View Herald TranscriptOct 17 2025, 6:57 AM

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

Peter added a parent task: T403005: [Placeholder] Automated browser testing.Oct 17 2025, 6:58 AM

Peter moved this task from Backlog to Ready to Refine on the Test Platform board.Oct 17 2025, 7:00 AM

Peter updated the task description. (Show Details)Oct 22 2025, 9:58 AM

Peter edited parent tasks, added: T407947: Improve visibility and detection of flakey WebdriverIO tests; removed: T403005: [Placeholder] Automated browser testing.

zeljkofilipin subscribed.Oct 27 2025, 5:43 PM

pwangai subscribed.Nov 11 2025, 2:44 AM

Peter added a project: Browser Test Platform.Nov 18 2025, 9:55 AM

Peter moved this task from Backlog to Prioritized on the Browser Test Platform board.Nov 18 2025, 11:20 AM

zeljkofilipin renamed this task from Investigate how to run webdriver.io/mediawiki on a slow machines/instances to easier find flakey tests to Investigate how to run webdriverio/mediawiki on a slow machines/instances to easier find flakey tests.Nov 20 2025, 3:05 PM

Peter renamed this task from Investigate how to run webdriverio/mediawiki on a slow machines/instances to easier find flakey tests to Investigate how to run webdriverio/mediawiki on a slow machine/instance to easier find flakey tests.Nov 20 2025, 5:12 PM

SDunlap assigned this task to pwangai.Nov 20 2025, 5:18 PM

SDunlap set the point value for this task to 5.

SDunlap moved this task from Ready to Refine to Refined (Discussed and Estimated) Backlog on the Test Platform board.

We discussed this in the meeting yesterday, let me write a suggestion how this can be done:

If you have a slow computer you can use this as a POC to see that we can get test to fail. Run core tests but also some tests from a extension (find someone that seems flakey, you can find extensions here.
Then the next step (or the first step if you don't have a slow computer ready) I would try to use Docker to slow down MediaWiki and or the container that runs the wdio test. Potentially this could be done in quickstart with some hacking.

If we can get tests to fail by making it slower, then we know that this could be a way forward. If we don't get them to fail, then we don't need to spend on this idea.

zeljkofilipin moved this task from Prioritized to Doing on the Browser Test Platform board.Nov 21 2025, 2:24 PM

pwangai moved this task from Refined (Discussed and Estimated) Backlog to Bashkent One 19 on the Test Platform board.Nov 24 2025, 2:41 PM

pwangai edited projects, added Test Platform (Bashkent One 19); removed Test Platform.

SDunlap edited projects, added Test Platform (Bashkent Two 20); removed Test Platform (Bashkent One 19).Dec 1 2025, 4:44 PM

pwangai moved this task from Backlog to In Progress on the Test Platform (Bashkent Two 20) board.Dec 2 2025, 4:04 AM

Using this config legend to test Core results on a local machine:

Config ID	CPU Cores	RAM (GB)
CI	2	2
C2	2	4
C3	2	6
C4	2	8
C5	4	2
C6	4	4
C7	4	6
C8	4	8
C9	6	2
C10	6	4
C11	6	6
C12	6	8
C13	8	2
C14	8	4
C15	8	6
C16	8	8

I get these results:

Test ID	Repo	Phase	Env Details	Config ID	CPU Cores	RAM (GB)	Max Instances	Total Tests	Passed	Failed	Duration	Max Instance 4
T001	Core	Quickstart	MacBook M1 Max	CI	2	2	1	10	9	1	13:13 min	8:54 min
T002	Core	Quickstart	MacBook M1 Max	C2	2	4	1	10	9	1	13:35 min	7:57 min
T003	Core	Quickstart	MacBook M1 Max	C3	2	6	1	10	9	1	13:09 min	7:29 min
T004	Core	Quickstart	MacBook M1 Max	C4	2	8	1	10	9	1	13:11 min	08:28 min
T005	Core	Quickstart	MacBook M1 Max	C5	4	2	1	10	10	0	8:07 min	4:13 min
T006	Core	Quickstart	MacBook M1 Max	C6	4	4	1	10	10	0	7:44 min	4:58 min
T007	Core	Quickstart	MacBook M1 Max	C7	4	6	1	10	10	0	7:43 min	3:55 min
T008	Core	Quickstart	MacBook M1 Max	C8	4	8	1	10	10	0	7:34 min	3:54 min
T009	Core	Quickstart	MacBook M1 Max	C9	6	2	1	10	10	0	08:04 min	3:52 min
T010	Core	Quickstart	MacBook M1 Max	C10	6	4	1	10	10	0	07:45 min	3:36 min
T011	Core	Quickstart	MacBook M1 Max	C11	6	6	1	10	10	0	7:28 min	3:35 min
T012	Core	Quickstart	MacBook M1 Max	C12	6	8	1	10	10	0	7:22 min	3:20 min
T013	Core	Quickstart	MacBook M1 Max	C13	8	2	1	10	10	0	7:53 min	3:53 min
T014	Core	Quickstart	MacBook M1 Max	C14	8	4	1	10	10	0	7:49 min	3:36 min
T015	Core	Quickstart	MacBook M1 Max	C15	8	6	1	10	10	0	7:37 min	3:33 min
T016	Core	Quickstart	MacBook M1 Max	C16	8	8	1	10	10	0	7:09 min	3:18 min

Below are the failures that happened when running Core test:

Test ID	Failed Test Name	Error Message	Error Type	CPU Cores	Ram	Config ID	Max Instance	Reproducible	Retry Result
T001	should be able to block a user	element (".cdx-menu-item--enabled") still not clickable after 10000ms	Timeout	2	2	C1	1 & 4	Yes	Fail
T002	should be able to block a user	element (".cdx-menu-item--enabled") still not clickable after 10000ms	Timeout	2	4	C2	1 & 4	Yes	Fail
T003	should be able to block a use	element (".cdx-menu-item--enabled") still not clickable after 10000ms	Timeout	2	6	C3	1 & 4	Yes	Fail
T004	should be able to block a use	element (".cdx-menu-item--enabled") still not clickable after 10000ms	Timeout	2	8	C4	1 & 4	Yes	Fail

TwoColConflict seems to be the most flaky extension according to this data. Running tests using the above config legend will help us see if we can get the tests to fail with the same common failures.

I created a script that automatically runs tests under different RAM and CPU configurations using Quickstart, with the ability to test various max instances. The script is not perfect, but from initial runs, I have noticed a few details.

For MediaWiki Core, with resources throttled down, selenium tests seem to be more stable. Even when running multiple max instances with limited resources, few errors were noticed. Here is a sample report generated from testing Core using the script instead of manually running the tests:

Selenium Test Results - Resource Configuration Analysis

Generated: 2025-12-06
Environment: MacBook M1
Test Path: Core tests
Max Instances Tested: 1,3,4

Test Summary

Test ID	Repo	Phase	Env Details	Config ID	CPU Cores	RAM (GB)	Total Tests	Passed	Max Instance: 1	Max Instance: 3	Max Instance: 4
T001	Core	Quickstart	MacBook M1	C2	2	4	10	10	03:53 min	03:17 min	03:23 min
T002	Core	Quickstart	MacBook M1	C3	2	6	10	10	03:38 min	03:24 min	03:17 min
T003	Core	Quickstart	MacBook M1	C4	2	8	10	10	03:40 min	03:14 min	02:42 min
T004	Core	Quickstart	MacBook M1	C6	4	4	10	10	02:28 min	01:48 min	02:23 min
T005	Core	Quickstart	MacBook M1	C7	4	6	10	10	02:25 min	02:10 min	01:44 min
T006	Core	Quickstart	MacBook M1	C8	4	8	10	10	02:28 min	02:02 min	01:45 min
T007	Core	Quickstart	MacBook M1	C10	6	4	10	10	02:22 min	01:38 min	01:58 min
T008	Core	Quickstart	MacBook M1	C11	6	6	10	10	02:56 min	01:34 min	01:30 min
T009	Core	Quickstart	MacBook M1	C12	6	8	10	10	02:41 min	02:06 min	01:16 min
T010	Core	Quickstart	MacBook M1	C14	8	4	10	10	02:18 min	01:52 min	01:30 min
T011	Core	Quickstart	MacBook M1	C15	8	6	10	10	02:24 min	01:19 min	02:22 min
T012	Core	Quickstart	MacBook M1	C16	8	8	10	10	02:24 min	01:15 min	01:41 min

Failure Details

Test ID	Failed Test Name	Error Message	Error Type	CPU Cores	Ram	Config ID	Max Instance	Reproducible	Retry Result
T011	Page.	Could not login: Unknown reason	Error	8	6	C15	4	Yes	Fail
T011	User.	Could not login: Unknown reason	Error	8	6	C15	4	Yes	Fail

Analysis

Total Configurations Tested: 12

For extensions, I tested out TwoColConflict, which is one of the repos that seem to have the most flaky tests. When running a max instance of 1, TwoColConflict Selenium tests seem to be stable with few to no errors, regardless of resources available. Example test run:

Selenium Test Results - Resource Configuration Analysis

Generated: 2025-12-06 15:41:15
Environment: MacBook M1
Test Path: extensions/TwoColConflict
Max Instances Tested: 1

Test Summary

Test ID	Repo	Phase	Env Details	Config ID	CPU Cores	RAM (GB)	Total Tests	Passed	Max Instance: 1
T001	TwoColConflict	Quickstart	MacBook M1	C2	2	4	7	7	03:36 min
T002	TwoColConflict	Quickstart	MacBook M1	C3	2	6	7	7	03:48 min
T003	TwoColConflict	Quickstart	MacBook M1	C4	2	8	7	7	03:42 min
T004	TwoColConflict	Quickstart	MacBook M1	C6	4	4	7	7	03:06 min
T005	TwoColConflict	Quickstart	MacBook M1	C7	4	6	7	7	03:04 min
T006	TwoColConflict	Quickstart	MacBook M1	C8	4	8	7	7	03:03 min
T007	TwoColConflict	Quickstart	MacBook M1	C10	6	4	7	7	03:03 min
T008	TwoColConflict	Quickstart	MacBook M1	C11	6	6	7	7	03:08 min
T009	TwoColConflict	Quickstart	MacBook M1	C12	6	8	7	7	03:02 min
T010	TwoColConflict	Quickstart	MacBook M1	C14	8	4	7	7	03:02 min
T011	TwoColConflict	Quickstart	MacBook M1	C15	8	6	7	7	03:02 min
T012	TwoColConflict	Quickstart	MacBook M1	C16	8	8	7	7	03:00 min

Analysis

Total Configurations Tested: 12

When the max instances value is increased beyond one for TwoColConflict, the tests become flaky, and we start seeing numerous errors. I also tested CampaignEvents, and the results were the same, tests pass with one max instance but fail with more than one.
We should investigate why Core Selenium tests remain stable when running multiple instances while extension tests fail under the same conditions. This may be due to a configuration difference, test structure difference, or even how the tests are written. If we resolve this, I believe we can significantly improve the robustness of the tests, which would allow us to enable multiple instances across all repositories and reduce wait times. The script generated report is a bit lengthy, but here are the TwoColConflict selenium test results for an example, no failures on 1 instance, but lots of failures on >1 instances:

Selenium Test Results - Resource Configuration Analysis

Generated: 2025-12-06 19:04:09
Environment: MacBook M1
Test Path: extensions/TwoColConflict
Max Instances Tested: 1,2,3,4

Test Summary

Test ID	Repo	Phase	Env Details	Config ID	CPU Cores	RAM (GB)	Total Tests	Passed	Max Instance: 1	Max Instance: 2	Max Instance: 3	Max Instance: 4
T001	TwoColConflict	Quickstart	MacBook M1	C2	2	4	7	7	03:44 min	05:47 min	04:10 min	04:55 min
T002	TwoColConflict	Quickstart	MacBook M1	C3	2	6	7	7	03:41 min	04:57 min	04:55 min	04:42 min
T003	TwoColConflict	Quickstart	MacBook M1	C4	2	8	7	7	03:42 min	05:15 min	04:47 min	05:00 min
T004	TwoColConflict	Quickstart	MacBook M1	C6	4	4	7	7	03:05 min	03:55 min	02:37 min	02:46 min
T005	TwoColConflict	Quickstart	MacBook M1	C7	4	6	7	7	03:04 min	04:32 min	03:51 min	03:47 min
T006	TwoColConflict	Quickstart	MacBook M1	C8	4	8	7	7	03:04 min	03:41 min	04:03 min	03:04 min
T007	TwoColConflict	Quickstart	MacBook M1	C10	6	4	7	7	03:03 min	03:49 min	04:18 min	03:37 min
T008	TwoColConflict	Quickstart	MacBook M1	C11	6	6	7	7	03:04 min	03:56 min	03:28 min	02:49 min
T009	TwoColConflict	Quickstart	MacBook M1	C12	6	8	7	7	03:02 min	03:36 min	03:37 min	03:11 min
T010	TwoColConflict	Quickstart	MacBook M1	C14	8	4	7	7	03:02 min	03:48 min	02:40 min	03:28 min
T011	TwoColConflict	Quickstart	MacBook M1	C15	8	6	7	7	03:01 min	04:37 min	04:09 min	03:18 min
T012	TwoColConflict	Quickstart	MacBook M1	C16	8	8	7	7	03:00 min	03:52 min	03:16 min	04:10 min

Failure Details

Test ID	Failed Test Name	Error Message	Error Type	CPU Cores	Ram	Config ID	Max Instance	Reproducible	Retry Result
T001	TwoColConflict collapse button.	Cannot submit login form	Error	2	4	C2	2	Yes	Fail
T001	TwoColConflict GuidedTour.	Cannot submit login form	Error	2	4	C2	4	No	Fail
T001	TwoColConflict.on talk page conflicts.	invalidjson: No valid JSON response	Error	2	4	C2	4	No	Fail
T002	TwoColConflict collapse button.	Cannot submit login form	Error	2	6	C3	2	Yes	Pass
T002	TwoColConflict.on talk page conflicts.stores correct merge when swapped and edited	Node is either not clickable or not an HTMLElement	Error	2	6	C3	2	Yes	Fail
T002	TwoColConflict.on talk page conflicts.	invalidjson: No valid JSON response	Error	2	6	C3	4	Yes	Fail
T002	TwoColConflict GuidedTour.on subsequent view.	Failed to wait for ext.TwoColConflict.SplitJs to be ready after 5000	Error	2	6	C3	4	Yes	Fail
T002	TwoColConflict GuidedTour.	invalidjson: No valid JSON response	Error	2	6	C3	4	No	Fail
T002	TwoColConflict save and preview.should save a resolved conflict successfully when another user edits a different section in the meantime	Failed to wait for mediawiki.base to be ready after 5000 ms.	Error	2	6	C3	4	No	Pass
T003	TwoColConflict EditUi.	Evaluation failed: http	Error	2	8	C4	2	Yes	Fail
T003	TwoColConflict without JavaScript.handles order selection on the talk page version correctly	invalidjson: No valid JSON response	Error	2	8	C4	4	No	Fail
T003	TwoColConflict save and preview.should be possible to edit and preview the left (	invalidjson: No valid JSON response	Error	2	8	C4	4	Yes	Fail
T003	TwoColConflict GuidedTour.on subsequent view.	invalidjson: No valid JSON response	Error	2	8	C4	4	No	Fail
T004	TwoColConflict EditUi.	Cannot submit login form	Error	4	4	C6	2	Yes	Pass
T004	TwoColConflict.	Can't call setValue on element with selector "#wpName1" because elem	Error	4	4	C6	3	Yes	Fail
T004	TwoColConflict GuidedTour.	Cannot submit login form	Error	4	4	C6	4	Yes	Fail
T004	TwoColConflict GuidedTour.	Evaluation failed: http	Error	4	4	C6	4	No	Fail
T004	TwoColConflict.labels change according to selected column	invalidjson: No valid JSON response	Error	4	4	C6	4	No	Pass
T004	TwoColConflict save and preview.should be possible to edit and preview the left (	invalidjson: No valid JSON response	Error	4	4	C6	4	Yes	Pass
T004	TwoColConflict save and preview.should save a resolved conflict successfully when another user edits a different section in the meantime	Failed to wait for mediawiki.base to be ready after 5000 ms.	Error	4	4	C6	4	No	Fail
T005	TwoColConflict EditUi.	Cannot submit login form	Error	4	6	C7	2	Yes	Fail
T005	TwoColConflict without JavaScript.handles order selection on the talk page version correctly	invalidjson: No valid JSON response	Error	4	6	C7	2	Yes	Fail
T005	TwoColConflict.shows the talk page screen on conflicts that also add new lines	invalidjson: No valid JSON response	Error	4	6	C7	2	Yes	Fail
T005	TwoColConflict.on talk page conflicts.	invalidjson: No valid JSON response	Error	4	6	C7	2	No	Fail
T005	TwoColConflict without JavaScript.handles order selection on the talk page version correctly	invalidjson: No valid JSON response	Error	4	6	C7	4	No	Fail
T005	TwoColConflict.editor should not decode html entities	invalidjson: No valid JSON response	Error	4	6	C7	4	Yes	Fail
T006	TwoColConflict EditUi.	Evaluation failed: http	Error	4	8	C8	2	Yes	Pass
T006	TwoColConflict save and preview.should save a resolved conflict successfully when another user edits a different section in the meantime	invalidjson: No valid JSON response	Error	4	8	C8	4	Yes	Pass
T006	TwoColConflict save and preview.should be possible to edit and preview the left (	invalidjson: No valid JSON response	Error	4	8	C8	4	No	Fail
T007	TwoColConflict EditUi.	Evaluation failed: http	Error	6	4	C10	2	Yes	Pass
T007	TwoColConflict without JavaScript.handles order selection on the talk page version correctly	invalidjson: No valid JSON response	Error	6	4	C10	2	Yes	Fail
T007	TwoColConflict save and preview.should trigger a new conflict when another user edits in the same lines in the meantime	invalidjson: No valid JSON response	Error	6	4	C10	4	Yes	Fail
T007	TwoColConflict without JavaScript.handles order selection on the talk page version correctly	invalidjson: No valid JSON response	Error	6	4	C10	4	Yes	Fail
T007	TwoColConflict.editor should not decode html entities	invalidjson: No valid JSON response	Error	6	4	C10	4	Yes	Fail
T008	TwoColConflict EditUi.	Cannot submit login form	Error	6	6	C11	2	Yes	Pass
T008	TwoColConflict GuidedTour.on subsequent view.	invalidjson: No valid JSON response	Error	6	6	C11	4	Yes	Pass
T009	TwoColConflict.on talk page conflicts.	invalidjson: No valid JSON response	Error	6	8	C12	4	Yes	Fail
T009	TwoColConflict GuidedTour.on subsequent view.	invalidjson: No valid JSON response	Error	6	8	C12	4	Yes	Fail
T009	TwoColConflict save and preview.should be possible to edit and preview the left (	internal_api_error_DBQueryError: [e6b410a045fd6ffdc57907ca] Exceptio	Error	6	8	C12	4	Yes	Fail
T009	TwoColConflict GuidedTour.on subsequent view.	invalidjson: No valid JSON response	Error	6	8	C12	4	No	Fail
T010	TwoColConflict collapse button.	Cannot submit login form	Error	8	4	C14	2	Yes	Fail
T011	TwoColConflict.editor should not decode html entities	invalidjson: No valid JSON response	Error	8	6	C15	4	Yes	Fail
T011	TwoColConflict save and preview.should save a resolved conflict successfully when another user edits a different section in the meantime	invalidjson: No valid JSON response	Error	8	6	C15	4	Yes	Pass
T011	TwoColConflict.editor should not decode html entities	invalidjson: No valid JSON response	Error	8	6	C15	4	No	Fail
T012	TwoColConflict collapse button.	Evaluation failed: http	Error	8	8	C16	2	Yes	Fail
T012	TwoColConflict save and preview.	Cannot submit login form	Error	8	8	C16	2	No	Fail
T012	TwoColConflict.on talk page conflicts.	invalidjson: No valid JSON response	Error	8	8	C16	4	No	Fail
T012	TwoColConflict.editor should not decode html entities	invalidjson: No valid JSON response	Error	8	8	C16	4	Yes	Fail