Page MenuHomePhabricator
Create Task
Maniphest T403822

AbuseFilter selenium tests are flaky
Open, Needs TriagePublic
Actions

Description

The selenium tests in AbuseFilter are flaky and caused CI to hang in https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-php81-selenium/28294/console

13:42:17 [0-0] RUNNING in chrome - /tests/selenium/specs/editingFilters.js
13:42:17 [0-2] RUNNING in chrome - /tests/selenium/specs/ViewList.basic.js
13:42:18 [0-1] RUNNING in chrome - /tests/selenium/specs/importingFilters.js
13:42:25 [0-2] PASSED in chrome - /tests/selenium/specs/ViewList.basic.js
13:42:40 [0-1] PASSED in chrome - /tests/selenium/specs/importingFilters.js
13:43:47 [0-0] Error in "Filter editing.Restoring an old version of a filter.edit can be saved (3)"
13:43:47 Error: Timeout of 60000ms exceeded. The execution in the test "Restoring an old version of a filter edit can be saved (3)" took too long. Try to reduce the run time or increase your timeout for test specs (https://webdriver.io/docs/timeouts). (/workspace/src/extensions/AbuseFilter/tests/selenium/specs/editingFilters.js)
13:43:47     at createTimeoutError (/workspace/src/extensions/AbuseFilter/node_modules/mocha/lib/errors.js:498:15)
13:43:47     at Runnable._timeoutError (/workspace/src/extensions/AbuseFilter/node_modules/mocha/lib/runnable.js:429:10)
13:43:47     at Timeout.<anonymous> (/workspace/src/extensions/AbuseFilter/node_modules/mocha/lib/runnable.js:244:24)
13:43:47     at listOnTimeout (node:internal/timers:581:17)
13:43:47     at process.processTimers (node:internal/timers:519:7)
13:44:47 [0-0] Error in "Filter editing.Restoring an old version of a filter.saved data is retained (3)"
13:44:47 Error: Timeout of 60000ms exceeded. The execution in the test "Restoring an old version of a filter saved data is retained (3)" took too long. Try to reduce the run time or increase your timeout for test specs (https://webdriver.io/docs/timeouts). (/workspace/src/extensions/AbuseFilter/tests/selenium/specs/editingFilters.js)
13:44:47     at createTimeoutError (/workspace/src/extensions/AbuseFilter/node_modules/mocha/lib/errors.js:498:15)
13:44:47     at Runnable._timeoutError (/workspace/src/extensions/AbuseFilter/node_modules/mocha/lib/runnable.js:429:10)
13:44:47     at Timeout.<anonymous> (/workspace/src/extensions/AbuseFilter/node_modules/mocha/lib/runnable.js:244:24)
13:44:47     at listOnTimeout (node:internal/timers:581:17)
13:44:47     at process.processTimers (node:internal/timers:519:7)
13:45:47 [0-0] Error in "Filter editing.CSRF protection."before all" hook for "a CSRF token is required to save the filter""
13:45:47 Error: Timeout of 60000ms exceeded. The execution in the test "CSRF protection "before all" hook for "a CSRF token is required to save the filter"" took too long. Try to reduce the run time or increase your timeout for test specs (https://webdriver.io/docs/timeouts). (/workspace/src/extensions/AbuseFilter/tests/selenium/specs/editingFilters.js)
13:45:47     at createTimeoutError (/workspace/src/extensions/AbuseFilter/node_modules/mocha/lib/errors.js:498:15)
13:45:47     at Runnable._timeoutError (/workspace/src/extensions/AbuseFilter/node_modules/mocha/lib/runnable.js:429:10)
13:45:47     at Timeout.<anonymous> (/workspace/src/extensions/AbuseFilter/node_modules/mocha/lib/runnable.js:244:24)
13:45:47     at listOnTimeout (node:internal/timers:581:17)
13:45:47     at process.processTimers (node:internal/timers:519:7)
13:46:47 [0-0] Error in "Filter editing.Trying to save a filter with bad data."before all" hook for "cannot save an empty filter""
13:46:47 Error: Timeout of 60000ms exceeded. The execution in the test "Trying to save a filter with bad data "before all" hook for "cannot save an empty filter"" took too long. Try to reduce the run time or increase your timeout for test specs (https://webdriver.io/docs/timeouts). (/workspace/src/extensions/AbuseFilter/tests/selenium/specs/editingFilters.js)
13:46:47     at createTimeoutError (/workspace/src/extensions/AbuseFilter/node_modules/mocha/lib/errors.js:498:15)
13:46:47     at Runnable._timeoutError (/workspace/src/extensions/AbuseFilter/node_modules/mocha/lib/runnable.js:429:10)
13:46:47     at Timeout.<anonymous> (/workspace/src/extensions/AbuseFilter/node_modules/mocha/lib/runnable.js:244:24)
13:46:47     at listOnTimeout (node:internal/timers:581:17)
13:46:47     at process.processTimers (node:internal/timers:519:7)
14:31:36 Build was aborted
14:31:36 Aborted by Dreamy Jazz
14:31:36 [PostBuildScript] - [INFO] Executing post build scripts.
14:31:36 [quibble-vendor-mysql-php81-selenium] $ /bin/bash -xe /tmp/jenkins8164863231035899447.sh

Acceptance criteria

  • The selenium tests are fixed, skipped, or replaced with other more reliable testing methods

Related Objects

Event Timeline

Dreamy_Jazz created this task.Sep 5 2025, 1:26 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 5 2025, 1:26 PM
Daimona subscribed.Sep 5 2025, 1:30 PM

I see this kind of timeouts all the time in tests that are sensitive to timeouts (mostly selenium and api-testing). My understanding is that there are times when CI is just excruciatingly slow, possibly due to other tests being run in parallel, and that in turn triggers timeouts of all sorts. I don't think the tests themselves are flaky or otherwise problematic, it's just the infrastructure. I don't know what can be done about it, though.

Dreamy_Jazz updated the task description. (Show Details)EditedSep 5 2025, 1:32 PM

Sure, but surely the build should have failed once the timeout was exceeded without having me to manually abort it over 40 minutes later?

This is part of the reason that Trust and Safety Product Team dropped all our selenium tests as they were being exceeding unreliable.

Daimona added a comment.Sep 5 2025, 1:35 PM
In T403822#11151964, @Dreamy_Jazz wrote:

Sure, but surely the build should have failed once the timeout was exceeded without having me to manually abort it over 40 minutes later?

Totally, but that's T389536, which I filed back in March. I don't think anyone really knows why it happens. A few people have thrown various ideas at the problem, but nothing worked. I think it could still be related to the CI workers being under pressure, but then I don't really know.

Dreamy_Jazz added a comment.Sep 19 2025, 3:00 PM

Happened again https://integration.wikimedia.org/ci/job/quibble-with-gated-extensions-selenium-php81/1040/console:

14:55:25 [0-0] RUNNING in chrome - /tests/selenium/specs/editingFilters.js
14:55:25 [0-1] RUNNING in chrome - /tests/selenium/specs/importingFilters.js
14:55:25 [0-2] RUNNING in chrome - /tests/selenium/specs/ViewList.basic.js
14:55:41 [0-2] PASSED in chrome - /tests/selenium/specs/ViewList.basic.js
14:56:05 [0-1] PASSED in chrome - /tests/selenium/specs/importingFilters.js
14:57:13 [0-0] Error in "Filter editing.Restoring an old version of a filter.edit can be saved (3)"
14:57:13 Error: Timeout of 60000ms exceeded. The execution in the test "Restoring an old version of a filter edit can be saved (3)" took too long. Try to reduce the run time or increase your timeout for test specs (https://webdriver.io/docs/timeouts). (/workspace/src/extensions/AbuseFilter/tests/selenium/specs/editingFilters.js)
14:57:13     at createTimeoutError (/workspace/src/extensions/AbuseFilter/node_modules/mocha/lib/errors.js:498:15)
14:57:13     at Runnable._timeoutError (/workspace/src/extensions/AbuseFilter/node_modules/mocha/lib/runnable.js:429:10)
14:57:13     at Timeout.<anonymous> (/workspace/src/extensions/AbuseFilter/node_modules/mocha/lib/runnable.js:244:24)
14:57:13     at listOnTimeout (node:internal/timers:581:17)
14:57:13     at process.processTimers (node:internal/timers:519:7)
14:58:13 [0-0] Error in "Filter editing.Restoring an old version of a filter.saved data is retained (3)"
14:58:13 Error: Timeout of 60000ms exceeded. The execution in the test "Restoring an old version of a filter saved data is retained (3)" took too long. Try to reduce the run time or increase your timeout for test specs (https://webdriver.io/docs/timeouts). (/workspace/src/extensions/AbuseFilter/tests/selenium/specs/editingFilters.js)
14:58:13     at createTimeoutError (/workspace/src/extensions/AbuseFilter/node_modules/mocha/lib/errors.js:498:15)
14:58:13     at Runnable._timeoutError (/workspace/src/extensions/AbuseFilter/node_modules/mocha/lib/runnable.js:429:10)
14:58:13     at Timeout.<anonymous> (/workspace/src/extensions/AbuseFilter/node_modules/mocha/lib/runnable.js:244:24)
14:58:13     at listOnTimeout (node:internal/timers:581:17)
14:58:13     at process.processTimers (node:internal/timers:519:7)
14:59:13 [0-0] Error in "Filter editing.CSRF protection."before all" hook for "a CSRF token is required to save the filter""
14:59:13 Error: Timeout of 60000ms exceeded. The execution in the test "CSRF protection "before all" hook for "a CSRF token is required to save the filter"" took too long. Try to reduce the run time or increase your timeout for test specs (https://webdriver.io/docs/timeouts). (/workspace/src/extensions/AbuseFilter/tests/selenium/specs/editingFilters.js)
14:59:13     at createTimeoutError (/workspace/src/extensions/AbuseFilter/node_modules/mocha/lib/errors.js:498:15)
14:59:13     at Runnable._timeoutError (/workspace/src/extensions/AbuseFilter/node_modules/mocha/lib/runnable.js:429:10)
14:59:13     at Timeout.<anonymous> (/workspace/src/extensions/AbuseFilter/node_modules/mocha/lib/runnable.js:244:24)
14:59:13     at listOnTimeout (node:internal/timers:581:17)
14:59:13     at process.processTimers (node:internal/timers:519:7)
15:00:13 [0-0] Error in "Filter editing.Trying to save a filter with bad data."before all" hook for "cannot save an empty filter""
15:00:13 Error: Timeout of 60000ms exceeded. The execution in the test "Trying to save a filter with bad data "before all" hook for "cannot save an empty filter"" took too long. Try to reduce the run time or increase your timeout for test specs (https://webdriver.io/docs/timeouts). (/workspace/src/extensions/AbuseFilter/tests/selenium/specs/editingFilters.js)
15:00:13     at createTimeoutError (/workspace/src/extensions/AbuseFilter/node_modules/mocha/lib/errors.js:498:15)
15:00:13     at Runnable._timeoutError (/workspace/src/extensions/AbuseFilter/node_modules/mocha/lib/runnable.js:429:10)
15:00:13     at Timeout.<anonymous> (/workspace/src/extensions/AbuseFilter/node_modules/mocha/lib/runnable.js:244:24)
15:00:13     at listOnTimeout (node:internal/timers:581:17)
15:00:13     at process.processTimers (node:internal/timers:519:7)
Dreamy_Jazz added a comment.Dec 19 2025, 10:38 AM

Happened again https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-php83-selenium/5348/console:

09:56:43 Execution of 3 workers started at 2025-12-19T09:56:43.957Z
09:56:43 
09:56:44 [0-2] RUNNING in chrome - /tests/selenium/specs/ViewList.basic.js
09:56:44 [0-0] RUNNING in chrome - /tests/selenium/specs/editingFilters.js
09:56:44 [0-1] RUNNING in chrome - /tests/selenium/specs/importingFilters.js
09:56:54 [0-2] PASSED in chrome - /tests/selenium/specs/ViewList.basic.js
09:57:27 [0-0] PASSED in chrome - /tests/selenium/specs/editingFilters.js
09:57:46 [0-1] Error in "When importing a filter."before all" hook for "the interface should be visible""
09:57:46 Error: Timeout of 60000ms exceeded. The execution in the test "When importing a filter "before all" hook for "the interface should be visible"" took too long. Try to reduce the run time or increase your timeout for test specs (https://webdriver.io/docs/timeouts). (/workspace/src/extensions/AbuseFilter/tests/selenium/specs/importingFilters.js)
09:57:46     at createTimeoutError (/workspace/src/extensions/AbuseFilter/node_modules/mocha/lib/errors.js:498:15)
09:57:46     at Runnable._timeoutError (/workspace/src/extensions/AbuseFilter/node_modules/mocha/lib/runnable.js:429:10)
09:57:46     at Timeout.<anonymous> (/workspace/src/extensions/AbuseFilter/node_modules/mocha/lib/runnable.js:244:24)
09:57:46     at listOnTimeout (node:internal/timers:581:17)
09:57:46     at process.processTimers (node:internal/timers:519:7)
10:38:19 Build was aborted
10:38:19 Aborted by Dreamy Jazz
10:38:19 [PostBuildScript] - [INFO] Executing post build scripts.
10:38:19 [quibble-vendor-mysql-php83-selenium] $ /bin/bash -xe /tmp/jenkins18381310958968140141.sh
10:38:19 + find log/ -name 'mw-debug-*.log' -exec gzip '{}' +
PostBuildScript
10:38:20 [PostBuildScript] - [INFO] Executing post build scripts.
10:38:20 [PostBuildScript] - [INFO] Build does not have any of the results [SUCCESS]. Did not execute build step #0.
10:38:20 [PostBuildScript] - [INFO] Executing post build scripts.
10:38:20 [PostBuildScript] - [INFO] Build does not have any of the results [SUCCESS]. Did not execute build step #0.
10:38:20 [PostBuildScript] - [INFO] Executing post build scripts.
10:38:20 [quibble-vendor-mysql-php83-selenium] $ /bin/bash -xe /tmp/jenkins8494875474578850578.sh
10:38:20 + echo 'Clearing /srv/jenkins/workspace/quibble-vendor-mysql-php83-selenium/cache'
10:38:20 Clearing /srv/jenkins/workspace/quibble-vendor-mysql-php83-selenium/cache
10:38:20 [quibble-vendor-mysql-php83-selenium] $ /bin/bash /tmp/jenkins15893939655626927882.sh
10:38:20 + set +x
10:38:20 + exec docker run --volume /srv/jenkins/workspace/quibble-vendor-mysql-php83-selenium/cache:/cache --security-opt seccomp=unconfined --init --rm --label jenkins.job=quibble-vendor-mysql-php83-selenium --label jenkins.build=5348 --env-file /dev/fd/63 docker-registry.wikimedia.org/releng/castor:0.4.0 clear
10:38:20 ++ set +x
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits