Page MenuHomePhabricator

Some GrowthExperiments Selenium tests are flaky
Closed, ResolvedPublic

Description

Some of the Selenium tests seem to be flaky. I ran into similar issues regularly lately.

See https://integration.wikimedia.org/ci/job/wmf-quibble-selenium-php74/49261/console
or https://integration.wikimedia.org/ci/job/wmf-quibble-selenium-php74/48774/console

10:31:37 [0-0] Error in "add image.desktop: user can view image info and image details"
10:31:37 Evaluation failed: notloggedin
10:31:37 Error: Evaluation failed: notloggedin
10:31:37     at ExecutionContext._evaluateInternal (/workspace/src/extensions/GrowthExperiments/node_modules/puppeteer-core/lib/cjs/puppeteer/common/ExecutionContext.js:221:19)
10:31:37     at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
10:31:37     at async ExecutionContext.evaluate (/workspace/src/extensions/GrowthExperiments/node_modules/puppeteer-core/lib/cjs/puppeteer/common/ExecutionContext.js:110:16)
10:31:37     at async ElementHandle.evaluate (/workspace/src/extensions/GrowthExperiments/node_modules/puppeteer-core/lib/cjs/puppeteer/common/JSHandle.js:107:16)
10:31:37     at async ElementHandle.$eval (/workspace/src/extensions/GrowthExperiments/node_modules/puppeteer-core/lib/cjs/puppeteer/common/JSHandle.js:810:24)
10:31:37     at async DevToolsDriver.executeScript (/workspace/src/extensions/GrowthExperiments/node_modules/devtools/build/commands/executeScript.js:39:20)
10:31:37     at async Browser.wrappedCommand (/workspace/src/extensions/GrowthExperiments/node_modules/devtools/build/devtoolsdriver.js:102:26)
10:31:37     at async AddImageArticlePage.setup (/workspace/src/extensions/GrowthExperiments/tests/selenium/pageobjects/addimage.article.page.js:84:3)
10:31:37     at async Context.<anonymous> (/workspace/src/extensions/GrowthExperiments/tests/selenium/specs/addimage.js:19:3)

and

11:32:29 [0-1] Error in "add link.link inspector can be used to accept/reject links and save an article."
11:32:29 Error: element (".mw-ge-recommendedLinkToolbarDialog") still not clickable after 30000ms
11:32:29     at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
11:32:29     at async AddLinkArticlePage.waitForDisplayedAndClickable (/workspace/src/extensions/GrowthExperiments/tests/selenium/pageobjects/addlink.article.page.js:113:3)
11:32:29     at async AddLinkArticlePage.waitForLinkInspector (/workspace/src/extensions/GrowthExperiments/tests/selenium/pageobjects/addlink.article.page.js:64:3)
11:32:29     at async Context.<anonymous> (/workspace/src/extensions/GrowthExperiments/tests/selenium/specs/addlink.js:32:3)

and

11:32:46 [0-1] Error in "add link.link inspector can be used to accept/reject links and save an article."
11:32:46 Error: Can't call getText on element with selector ".se-card-title" because element wasn't found
11:32:46     at async Context.<anonymous> (/workspace/src/extensions/GrowthExperiments/tests/selenium/specs/addlink.js:20:23)

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald Transcript
kostajh subscribed.

It looks like the content editable surface has changed so browser is blocked from exiting the page. I'm not sure why that only happens sometimes. It seems like a bug in webdriverio.

Change 921608 had a related patch set uploaded (by Kosta Harlan; author: Bartosz Dziewoński):

[mediawiki/extensions/GrowthExperiments@master] Temporarily disable failing tests

https://gerrit.wikimedia.org/r/921608

Change 921608 merged by Bartosz Dziewoński:

[mediawiki/extensions/GrowthExperiments@master] Temporarily disable failing tests

https://gerrit.wikimedia.org/r/921608

The desktop version can fail as well:
https://integration.wikimedia.org/ci/job/wmf-quibble-selenium-php74-docker/75476/console

22:34:44 [Chrome 90.0.4430.212 linux #0-0] Running: Chrome (v90.0.4430.212) on linux
22:34:44 [Chrome 90.0.4430.212 linux #0-0] Session ID: 474704fb-e1c1-4f67-8e0c-d14515444010
22:34:44 [Chrome 90.0.4430.212 linux #0-0]
22:34:44 [Chrome 90.0.4430.212 linux #0-0] » /tests/selenium/specs/addimage.js
22:34:44 [Chrome 90.0.4430.212 linux #0-0] add image
22:34:44 [Chrome 90.0.4430.212 linux #0-0]    ✖ desktop: user can view image info and image details
22:34:44 [Chrome 90.0.4430.212 linux #0-0]    - mobile: user can close the image suggestion UI
22:34:44 [Chrome 90.0.4430.212 linux #0-0]
22:34:44 [Chrome 90.0.4430.212 linux #0-0] 1 failing (12s)
22:34:44 [Chrome 90.0.4430.212 linux #0-0] 1 skipped
22:34:44 [Chrome 90.0.4430.212 linux #0-0]
22:34:44 [Chrome 90.0.4430.212 linux #0-0] 1) add image desktop: user can view image info and image details
22:34:44 [Chrome 90.0.4430.212 linux #0-0] element (".oo-ui-messageDialog-message") still not displayed after 5000ms
22:34:44 [Chrome 90.0.4430.212 linux #0-0] Error: element (".oo-ui-messageDialog-message") still not displayed after 5000ms
22:34:44 [Chrome 90.0.4430.212 linux #0-0]     at /workspace/src/extensions/GrowthExperiments/node_modules/webdriverio/build/commands/browser/waitUntil.js:66:23

https://integration.wikimedia.org/ci/job/wmf-quibble-selenium-php81-docker/16276/console

21:26:42 [Chrome 90.0.4430.212 linux #0-0] Running: Chrome (v90.0.4430.212) on linux
21:26:42 [Chrome 90.0.4430.212 linux #0-0] Session ID: abc381db-c582-4555-b538-239a0cc52dd7
21:26:42 [Chrome 90.0.4430.212 linux #0-0]
21:26:42 [Chrome 90.0.4430.212 linux #0-0] » /tests/selenium/specs/addimage.js
21:26:42 [Chrome 90.0.4430.212 linux #0-0] add image
21:26:42 [Chrome 90.0.4430.212 linux #0-0]    ✖ desktop: user can view image info and image details
21:26:42 [Chrome 90.0.4430.212 linux #0-0]    - mobile: user can close the image suggestion UI
21:26:42 [Chrome 90.0.4430.212 linux #0-0]
21:26:42 [Chrome 90.0.4430.212 linux #0-0] 1 failing (13.6s)
21:26:42 [Chrome 90.0.4430.212 linux #0-0] 1 skipped
21:26:42 [Chrome 90.0.4430.212 linux #0-0]
21:26:42 [Chrome 90.0.4430.212 linux #0-0] 1) add image desktop: user can view image info and image details
21:26:42 [Chrome 90.0.4430.212 linux #0-0] Evaluation failed: notloggedin
21:26:42 [Chrome 90.0.4430.212 linux #0-0] Evaluation failed: notloggedin
21:26:42 [Chrome 90.0.4430.212 linux #0-0] Error: Evaluation failed: notloggedin
21:26:42 [Chrome 90.0.4430.212 linux #0-0]     at ExecutionContext._evaluateInternal (/workspace/src/extensions/GrowthExperiments/node_modules/puppeteer-core/lib/cjs/puppeteer/common/ExecutionContext.js:221:19)
21:26:42 [Chrome 90.0.4430.212 linux #0-0]     at processTicksAndRejections (node:internal/process/task_queues:96:5)
21:26:42 [Chrome 90.0.4430.212 linux #0-0]     at async ExecutionContext.evaluate (/workspace/src/extensions/GrowthExperiments/node_modules/puppeteer-core/lib/cjs/puppeteer/common/ExecutionContext.js:110:16)
21:26:42 [Chrome 90.0.4430.212 linux #0-0]     at async ElementHandle.evaluate (/workspace/src/extensions/GrowthExperiments/node_modules/puppeteer-core/lib/cjs/puppeteer/common/JSHandle.js:107:16)
21:26:42 [Chrome 90.0.4430.212 linux #0-0]     at async ElementHandle.$eval (/workspace/src/extensions/GrowthExperiments/node_modules/puppeteer-core/lib/cjs/puppeteer/common/JSHandle.js:810:24)
21:26:42 [Chrome 90.0.4430.212 linux #0-0]     at async DevToolsDriver.executeScript (/workspace/src/extensions/GrowthExperiments/node_modules/devtools/build/commands/executeScript.js:39:20)
21:26:42 [Chrome 90.0.4430.212 linux #0-0]     at async Browser.wrappedCommand (/workspace/src/extensions/GrowthExperiments/node_modules/devtools/build/devtoolsdriver.js:102:26)
21:26:42 [Chrome 90.0.4430.212 linux #0-0]     at async Browser.wrapCommandFn (/workspace/src/extensions/GrowthExperiments/node_modules/@wdio/utils/build/shim.js:137:29)
21:26:42 [Chrome 90.0.4430.212 linux #0-0]     at async Browser.wrapCommandFn (/workspace/src/extensions/GrowthExperiments/node_modules/@wdio/utils/build/shim.js:137:29)
21:26:42 [Chrome 90.0.4430.212 linux #0-0]     at async AddImageArticlePage.setup (/workspace/src/extensions/GrowthExperiments/tests/selenium/pageobjects/addimage.article.page.js:50:3)
21:26:42 [Chrome 90.0.4430.212 linux #0-0]     at async Context.<anonymous> (/workspace/src/extensions/GrowthExperiments/tests/selenium/specs/addimage.js:19:3)

https://integration.wikimedia.org/ci/job/wmf-quibble-selenium-php74/29156/console

[0-0] Error in "add image.desktop: user can view image info and image details"
Evaluation failed: notloggedin
Error: Evaluation failed: notloggedin
    at ExecutionContext._evaluateInternal (/workspace/src/extensions/GrowthExperiments/node_modules/puppeteer-core/lib/cjs/puppeteer/common/ExecutionContext.js:221:19)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async ExecutionContext.evaluate (/workspace/src/extensions/GrowthExperiments/node_modules/puppeteer-core/lib/cjs/puppeteer/common/ExecutionContext.js:110:16)
    at async ElementHandle.evaluate (/workspace/src/extensions/GrowthExperiments/node_modules/puppeteer-core/lib/cjs/puppeteer/common/JSHandle.js:107:16)
    at async ElementHandle.$eval (/workspace/src/extensions/GrowthExperiments/node_modules/puppeteer-core/lib/cjs/puppeteer/common/JSHandle.js:810:24)
    at async DevToolsDriver.executeScript (/workspace/src/extensions/GrowthExperiments/node_modules/devtools/build/commands/executeScript.js:39:20)
    at async Browser.wrappedCommand (/workspace/src/extensions/GrowthExperiments/node_modules/devtools/build/devtoolsdriver.js:102:26)
    at async Browser.wrapCommandFn (/workspace/src/extensions/GrowthExperiments/node_modules/@wdio/utils/build/shim.js:137:29)
    at async Browser.wrapCommandFn (/workspace/src/extensions/GrowthExperiments/node_modules/@wdio/utils/build/shim.js:137:29)
    at async AddImageArticlePage.setup (/workspace/src/extensions/GrowthExperiments/tests/selenium/pageobjects/addimage.article.page.js:84:3)
    at async Context.<anonymous> (/workspace/src/extensions/GrowthExperiments/tests/selenium/specs/addimage.js:19:3)
[0-0] RETRYING after 3s in chrome - /tests/selenium/specs/addimage.js
[0-0] RUNNING in chrome - /tests/selenium/specs/addimage.js
[0-0] Error in "add image.desktop: user can view image info and image details"
Evaluation failed: notloggedin
Error: Evaluation failed: notloggedin
    at ExecutionContext._evaluateInternal (/workspace/src/extensions/GrowthExperiments/node_modules/puppeteer-core/lib/cjs/puppeteer/common/ExecutionContext.js:221:19)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async ExecutionContext.evaluate (/workspace/src/extensions/GrowthExperiments/node_modules/puppeteer-core/lib/cjs/puppeteer/common/ExecutionContext.js:110:16)
    at async ElementHandle.evaluate (/workspace/src/extensions/GrowthExperiments/node_modules/puppeteer-core/lib/cjs/puppeteer/common/JSHandle.js:107:16)
    at async ElementHandle.$eval (/workspace/src/extensions/GrowthExperiments/node_modules/puppeteer-core/lib/cjs/puppeteer/common/JSHandle.js:810:24)
    at async DevToolsDriver.executeScript (/workspace/src/extensions/GrowthExperiments/node_modules/devtools/build/commands/executeScript.js:39:20)
    at async Browser.wrappedCommand (/workspace/src/extensions/GrowthExperiments/node_modules/devtools/build/devtoolsdriver.js:102:26)
    at async Browser.wrapCommandFn (/workspace/src/extensions/GrowthExperiments/node_modules/@wdio/utils/build/shim.js:137:29)
    at async Browser.wrapCommandFn (/workspace/src/extensions/GrowthExperiments/node_modules/@wdio/utils/build/shim.js:137:29)
    at async AddImageArticlePage.setup (/workspace/src/extensions/GrowthExperiments/tests/selenium/pageobjects/addimage.article.page.js:84:3)
    at async Context.<anonymous> (/workspace/src/extensions/GrowthExperiments/tests/selenium/specs/addimage.js:19:3)
[0-0] RETRYING after 3s in chrome - /tests/selenium/specs/addimage.js (1 retries)
[0-0] RUNNING in chrome - /tests/selenium/specs/addimage.js
[0-0] Error in "add image.desktop: user can view image info and image details"
Evaluation failed: notloggedin
Error: Evaluation failed: notloggedin
    at ExecutionContext._evaluateInternal (/workspace/src/extensions/GrowthExperiments/node_modules/puppeteer-core/lib/cjs/puppeteer/common/ExecutionContext.js:221:19)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async ExecutionContext.evaluate (/workspace/src/extensions/GrowthExperiments/node_modules/puppeteer-core/lib/cjs/puppeteer/common/ExecutionContext.js:110:16)
    at async ElementHandle.evaluate (/workspace/src/extensions/GrowthExperiments/node_modules/puppeteer-core/lib/cjs/puppeteer/common/JSHandle.js:107:16)
    at async ElementHandle.$eval (/workspace/src/extensions/GrowthExperiments/node_modules/puppeteer-core/lib/cjs/puppeteer/common/JSHandle.js:810:24)
    at async DevToolsDriver.executeScript (/workspace/src/extensions/GrowthExperiments/node_modules/devtools/build/commands/executeScript.js:39:20)
    at async Browser.wrappedCommand (/workspace/src/extensions/GrowthExperiments/node_modules/devtools/build/devtoolsdriver.js:102:26)
    at async Browser.wrapCommandFn (/workspace/src/extensions/GrowthExperiments/node_modules/@wdio/utils/build/shim.js:137:29)
    at async Browser.wrapCommandFn (/workspace/src/extensions/GrowthExperiments/node_modules/@wdio/utils/build/shim.js:137:29)
    at async AddImageArticlePage.setup (/workspace/src/extensions/GrowthExperiments/tests/selenium/pageobjects/addimage.article.page.js:84:3)
    at async Context.<anonymous> (/workspace/src/extensions/GrowthExperiments/tests/selenium/specs/addimage.js:19:3)
[0-0] FAILED in chrome - /tests/selenium/specs/addimage.js (2 retries)

Change #1089338 had a related patch set uploaded (by Jared Blumer; author: Jared Blumer):

[mediawiki/extensions/GrowthExperiments@master] Remove skipped tests in addImage.js and delete homepage.js

https://gerrit.wikimedia.org/r/1089338

Change #1089338 abandoned by Zfilipin:

[mediawiki/extensions/GrowthExperiments@master] Remove skipped tests in addImage.js and delete homepage.js

Reason:

Abandoning since the test will be moved to cypress.

https://gerrit.wikimedia.org/r/1089338

https://integration.wikimedia.org/ci/job/wmf-quibble-selenium-php74/46146/console

00:11:20.036 [Chrome 90.0.4430.212 linux #0-0] 1) add image desktop: user can view image info and image details
00:11:20.036 [Chrome 90.0.4430.212 linux #0-0] Evaluation failed: notloggedin
00:11:20.036 [Chrome 90.0.4430.212 linux #0-0] Evaluation failed: notloggedin
00:11:20.036 [Chrome 90.0.4430.212 linux #0-0] Error: Evaluation failed: notloggedin
00:11:20.036 [Chrome 90.0.4430.212 linux #0-0]     at ExecutionContext._evaluateInternal (/workspace/src/extensions/GrowthExperiments/node_modules/puppeteer-core/lib/cjs/puppeteer/common/ExecutionContext.js:221:19)
00:11:20.036 [Chrome 90.0.4430.212 linux #0-0]     at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
00:11:20.037 [Chrome 90.0.4430.212 linux #0-0]     at async ExecutionContext.evaluate (/workspace/src/extensions/GrowthExperiments/node_modules/puppeteer-core/lib/cjs/puppeteer/common/ExecutionContext.js:110:16)
00:11:20.037 [Chrome 90.0.4430.212 linux #0-0]     at async ElementHandle.evaluate (/workspace/src/extensions/GrowthExperiments/node_modules/puppeteer-core/lib/cjs/puppeteer/common/JSHandle.js:107:16)
00:11:20.037 [Chrome 90.0.4430.212 linux #0-0]     at async ElementHandle.$eval (/workspace/src/extensions/GrowthExperiments/node_modules/puppeteer-core/lib/cjs/puppeteer/common/JSHandle.js:810:24)
00:11:20.037 [Chrome 90.0.4430.212 linux #0-0]     at async DevToolsDriver.executeScript (/workspace/src/extensions/GrowthExperiments/node_modules/devtools/build/commands/executeScript.js:39:20)
00:11:20.037 [Chrome 90.0.4430.212 linux #0-0]     at async Browser.wrappedCommand (/workspace/src/extensions/GrowthExperiments/node_modules/devtools/build/devtoolsdriver.js:102:26)
00:11:20.037 [Chrome 90.0.4430.212 linux #0-0]     at async AddImageArticlePage.setup (/workspace/src/extensions/GrowthExperiments/tests/selenium/pageobjects/addimage.article.page.js:84:3)
00:11:20.038 [Chrome 90.0.4430.212 linux #0-0]     at async Context.<anonymous> (/workspace/src/extensions/GrowthExperiments/tests/selenium/specs/addimage.js:19:3)

https://integration.wikimedia.org/ci/job/wmf-quibble-selenium-php74/49261/console

10:31:37 [0-0] Error in "add image.desktop: user can view image info and image details"
10:31:37 Evaluation failed: notloggedin
10:31:37 Error: Evaluation failed: notloggedin
10:31:37     at ExecutionContext._evaluateInternal (/workspace/src/extensions/GrowthExperiments/node_modules/puppeteer-core/lib/cjs/puppeteer/common/ExecutionContext.js:221:19)
10:31:37     at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
10:31:37     at async ExecutionContext.evaluate (/workspace/src/extensions/GrowthExperiments/node_modules/puppeteer-core/lib/cjs/puppeteer/common/ExecutionContext.js:110:16)
10:31:37     at async ElementHandle.evaluate (/workspace/src/extensions/GrowthExperiments/node_modules/puppeteer-core/lib/cjs/puppeteer/common/JSHandle.js:107:16)
10:31:37     at async ElementHandle.$eval (/workspace/src/extensions/GrowthExperiments/node_modules/puppeteer-core/lib/cjs/puppeteer/common/JSHandle.js:810:24)
10:31:37     at async DevToolsDriver.executeScript (/workspace/src/extensions/GrowthExperiments/node_modules/devtools/build/commands/executeScript.js:39:20)
10:31:37     at async Browser.wrappedCommand (/workspace/src/extensions/GrowthExperiments/node_modules/devtools/build/devtoolsdriver.js:102:26)
10:31:37     at async AddImageArticlePage.setup (/workspace/src/extensions/GrowthExperiments/tests/selenium/pageobjects/addimage.article.page.js:84:3)
10:31:37     at async Context.<anonymous> (/workspace/src/extensions/GrowthExperiments/tests/selenium/specs/addimage.js:19:3)

and

11:32:29 [0-1] Error in "add link.link inspector can be used to accept/reject links and save an article."
11:32:29 Error: element (".mw-ge-recommendedLinkToolbarDialog") still not clickable after 30000ms
11:32:29     at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
11:32:29     at async AddLinkArticlePage.waitForDisplayedAndClickable (/workspace/src/extensions/GrowthExperiments/tests/selenium/pageobjects/addlink.article.page.js:113:3)
11:32:29     at async AddLinkArticlePage.waitForLinkInspector (/workspace/src/extensions/GrowthExperiments/tests/selenium/pageobjects/addlink.article.page.js:64:3)
11:32:29     at async Context.<anonymous> (/workspace/src/extensions/GrowthExperiments/tests/selenium/specs/addlink.js:32:3)

and

11:32:46 [0-1] Error in "add link.link inspector can be used to accept/reject links and save an article."
11:32:46 Error: Can't call getText on element with selector ".se-card-title" because element wasn't found
11:32:46     at async Context.<anonymous> (/workspace/src/extensions/GrowthExperiments/tests/selenium/specs/addlink.js:20:23)
WMDE-Fisch renamed this task from Selenium test "add image.mobile: user can close the image suggestion UI" is flaky to Some GrowthExperiments Selenium tests are flaky.Jan 15 2025, 10:47 AM
Michael claimed this task.
Michael subscribed.

GrowthExperiments no longer has any Selenium tests 🎉