Page MenuHomePhabricator

Tone Check "crashes" sometimes when interacting with decline survey on mobile
Open, HighPublic

Description

It turns out that one of our new Revise Tone browser tests is flaky in CI.

Example: https://integration.wikimedia.org/ci/job/quibble-with-gated-extensions-selenium-php81/5206/

From looking at the log and the video (see below), it appears that the Edit Check just disappears after selecting one of the options on the decline-survey on mobile, before clicking "Submit".

00:13:58.384         cy:command ✔  contains	a, Decline
00:13:58.384         cy:command ✔  click	
00:13:58.384         cy:command ✔  get	.ve-ui-editCheckActionWidget
00:13:58.384         cy:command ✔  find	label
00:13:58.384         cy:command ✔  first	
00:13:58.384         cy:command ✔  click	
00:13:58.384         cy:command ✔  wait	250
00:13:58.384         cy:command ✔  get	.ve-ui-editCheckActionWidget
00:13:58.384         cy:command ✔  contains	button, Submit
00:13:58.384         cy:command ✘  assert	expected **subject** not to be **disabled**
00:13:58.384                       Actual: 	"subject"
00:13:58.384                       Expected: 	"disabled"
00:13:58.384 
00:13:58.384 
00:14:15.158       ✓ Shows the Revise Tone Edit Check and tags edits (16420ms)
00:14:15.159 
00:14:15.189 
00:14:15.189   3 passing (1m)
00:14:15.189   1 failing
00:14:15.189 
00:14:15.189   1) Revise Tone
00:14:15.189        On mobile
00:14:15.189          Closes the Editor when declining Edits and suggests a new task:
00:14:15.189      CypressError: Timed out retrying after 20000ms: `cy.contains()` failed because it requires a DOM element, window or document.
00:14:15.189 
00:14:15.189 No elements in the current DOM matched your query:
00:14:15.189 
00:14:15.189   > cy.get(.ve-ui-editCheckActionWidget)
00:14:15.189       at isElement (http://127.0.0.1:9413/__cypress/runner/cypress_runner.js:148556:62)
00:14:15.189       at validateType (http://127.0.0.1:9413/__cypress/runner/cypress_runner.js:148414:14)
00:14:15.189       at Object.isType (http://127.0.0.1:9413/__cypress/runner/cypress_runner.js:148445:7)
00:14:15.189       at <unknown> (http://127.0.0.1:9413/__cypress/runner/cypress_runner.js:135693:22)
00:14:15.190       at Object.subjectFn (http://127.0.0.1:9413/__cypress/runner/cypress_runner.js:147466:16)
00:14:15.190       at $Cy.verifyUpcomingAssertions (http://127.0.0.1:9413/__cypress/runner/cypress_runner.js:146788:31)
00:14:15.190       at onRetry (http://127.0.0.1:9413/__cypress/runner/cypress_runner.js:147457:15)
00:14:15.190       at tryCatcher (http://127.0.0.1:9413/__cypress/runner/cypress_runner.js:1777:23)
00:14:15.190       at Promise.attempt.Promise.try (http://127.0.0.1:9413/__cypress/runner/cypress_runner.js:4285:29)
00:14:15.190       at whenStable (http://127.0.0.1:9413/__cypress/runner/cypress_runner.js:147344:68)
00:14:15.190       at <unknown> (http://127.0.0.1:9413/__cypress/runner/cypress_runner.js:147285:14)
00:14:15.190       at tryCatcher (http://127.0.0.1:9413/__cypress/runner/cypress_runner.js:1777:23)
00:14:15.190       at Promise._settlePromiseFromHandler (http://127.0.0.1:9413/__cypress/runner/cypress_runner.js:1489:31)
00:14:15.190       at Promise._settlePromise (http://127.0.0.1:9413/__cypress/runner/cypress_runner.js:1546:18)
00:14:15.190       at Promise._settlePromise0 (http://127.0.0.1:9413/__cypress/runner/cypress_runner.js:1591:10)
00:14:15.190       at Promise._settlePromises (http://127.0.0.1:9413/__cypress/runner/cypress_runner.js:1671:18)
00:14:15.190       at Promise._fulfill (http://127.0.0.1:9413/__cypress/runner/cypress_runner.js:1615:18)
00:14:15.190       at <unknown> (http://127.0.0.1:9413/__cypress/runner/cypress_runner.js:5420:46)
00:14:15.190   From Your Spec Code:
00:14:15.190       at Context.eval (webpack://GrowthExperiments/./cypress/e2e/ReviseTone.cy.ts:93:75)
00:14:15.190

And the video: https://integration.wikimedia.org/ci/job/quibble-with-gated-extensions-selenium-php81/5206/artifact/log/GrowthExperiments/ReviseTone.cy.ts.mp4 (at around 0:34)

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

Change #1195752 had a related patch set uploaded (by Michael Große; author: Michael Große):

[mediawiki/extensions/GrowthExperiments@master] tests(Cypress): skip flaky Revise Tone test

https://gerrit.wikimedia.org/r/1195752

Change #1195752 merged by jenkins-bot:

[mediawiki/extensions/GrowthExperiments@master] tests(Cypress): skip flaky Revise Tone test

https://gerrit.wikimedia.org/r/1195752

Michael triaged this task as Medium priority.Oct 15 2025, 12:17 PM
Michael moved this task from Inbox to Backlog on the Growth-Team board.

Despite the skip, I still saw a test failure, now from the test on line 100: https://integration.wikimedia.org/ci/job/quibble-with-gated-extensions-selenium-php81/5570/console

+1. Seen this on at least two CheckUser patches

Change #1196398 had a related patch set uploaded (by Michael Große; author: Michael Große):

[mediawiki/extensions/GrowthExperiments@master] tests(Cypress): fix Revise Tone flakiness by intercepting model request

https://gerrit.wikimedia.org/r/1196398

That's a different test in that file. It seems that sometimes it edits the wrong paragraph? 🤨

Anyway, I added a patch that intercepts the request to the model and stubs an all-clear response. Should have done that from the start.

That's a different test in that file.

Thanks for the context (I had just searched for existing tasks with ReviseTone.cy.ts mentioned and saw this task)

https://integration.wikimedia.org/ci/job/quibble-with-gated-extensions-selenium-php81/6338/console

16:17:26   1) Revise Tone
16:17:26        On mobile
16:17:26          Shows the Revise Tone Edit Check and tags edits:
16:17:26      AssertionError: Timed out retrying after 20000ms: Expected to find element: `.ve-ui-mwSaveDialog .oo-ui-processDialog-actions-primary`, but never found it.
16:17:26       at Context.eval (webpack://GrowthExperiments/./cypress/e2e/ReviseTone.cy.ts:116:72)

Change #1197223 had a related patch set uploaded (by Michael Große; author: Michael Große):

[mediawiki/extensions/GrowthExperiments@master] tests(Cypress): Skip second Tevise Tone mobile browser test

https://gerrit.wikimedia.org/r/1197223

Change #1197223 merged by jenkins-bot:

[mediawiki/extensions/GrowthExperiments@master] tests(Cypress): Skip second Revise Tone mobile browser test

https://gerrit.wikimedia.org/r/1197223

https://integration.wikimedia.org/ci/job/quibble-with-gated-extensions-selenium-php81/6977/console

How many of these are broken? :(

23:29:50 ────────────────────────────────────────────────────────────────────────────────────────────────────
23:29:50                                                                                                     
23:29:50   Running:  ReviseTone.cy.ts                                                                (6 of 8)
23:29:53 
23:29:53 
23:29:53   Revise Tone
23:29:56     On desktop
23:30:22       1) Closes the Editor when declining Edits and suggests a new task
23:30:22         cy:command ✔  visit	/index.php -> 301: http://127.0.0.1:9413/index.php?title=Main_Page
23:30:22         cy:command ✔  window	
23:30:22         cy:command ✔  assert	expected **<window>** to have property **mw**
23:30:22         cy:command ✔  window	
23:30:22         cy:command ✔  its	.mw.Api
23:30:22         cy:command ✔  assert	expected **[Function]** to exist
23:30:22         cy:command ✔  window	
23:30:22             cy:xhr ➟  POST http://127.0.0.1:9413/api.php
23:30:22                       Status: 200
23:30:22             cy:xhr ➟  POST http://127.0.0.1:9413/api.php
23:30:22                       Status: 200
23:30:22         cy:command ✔  visit	index.php?title=Special:Homepage
23:30:22         cy:command ✔  get	.se-card-title
23:30:22         cy:command ✔  assert	expected **<h3.se-card-title>** to have text **Kristallsee**
23:30:22                       Actual: 	"Kristallsee"
23:30:22                       Expected: 	"Kristallsee"
23:30:22         cy:command ✔  get	a.se-card-content
23:30:22         cy:command ✔  assert	expected **<a.se-card-content>** not to have attribute **href** with the value **#**
23:30:22          cons:warn ❖  This page is using the deprecated ResourceLoader module "moment".
23:30:22                       [1.44] Use mediawiki.DateFormatter or native Intl function instead. See https://phabricator.wikimedia.org/T146798
23:30:22          cons:info ✱  You are running a development build of Vue.
23:30:22                       Make sure to use the production build (*.prod.js) when deploying for production.
23:30:22             cy:xhr ➟  POST http://127.0.0.1:9413/rest.php/growthexperiments/v0/user-impact/%2333?lang=en
23:30:22                       Status: 200
23:30:22         cy:command ✔  get	a.se-card-content
23:30:22         cy:command ✔  click	
23:30:22             cy:xhr ❖  GET http://127.0.0.1:9413/api/rest_v1/page/summary/Kristallsee
23:30:22                       Status: 404
23:30:22             cy:xhr ❖  GET https://wikimedia.org/api/rest_v1/metrics/pageviews/per-article/127.0.0.1/all-access/user/Kristallsee/daily/20250820/20251019
23:30:22                       Status: 404
23:30:22         cy:command ✔  page load	--page loaded--
23:30:22         cy:command ✔  new url	http://127.0.0.1:9413/index.php?title=Kristallsee&getasktype=revise-tone&gesuggestededit=1&geclickid=mrif8irrosp7lbvvjq8ccnafudpkgtb2&genewcomertasktoken=kb6o826apeqlhp7skdu7oosm4762i0t6&veaction=edit&section=all
23:30:22         cy:command ✔  get	.ve-ui-editCheckActionWidget
23:30:22         cy:command ✔  new url	http://127.0.0.1:9413/index.php?title=Kristallsee&getasktype=revise-tone&gesuggestededit=1&genewcomertasktoken=kb6o826apeqlhp7skdu7oosm4762i0t6&veaction=edit&section=all
23:30:22         cy:command ✔  new url	http://127.0.0.1:9413/index.php?title=Kristallsee&gesuggestededit=1&genewcomertasktoken=kb6o826apeqlhp7skdu7oosm4762i0t6&veaction=edit&section=all
23:30:22         cy:command ✔  new url	http://127.0.0.1:9413/index.php?title=Kristallsee&gesuggestededit=1&veaction=edit&section=all
23:30:22             cy:xhr ➟  POST http://127.0.0.1:9413/api.php
23:30:22                       Status: 200
23:30:22             cy:xhr ➟  GET http://127.0.0.1:9413/api.php?action=visualeditor&format=json&paction=parse&page=Kristallsee&uselang=en&editintro=&preload=&preloadparams=&formatversion=2
23:30:22                       Status: 200
23:30:22          cons:warn ❖  This page is using the deprecated ResourceLoader module "moment".
23:30:22                       [1.44] Use mediawiki.DateFormatter or native Intl function instead. See https://phabricator.wikimedia.org/T146798
23:30:22          cons:info ✱  You are running a development build of Vue.
23:30:22                       Make sure to use the production build (*.prod.js) when deploying for production.
23:30:22         cy:command ✔  new url	http://127.0.0.1:9413/index.php?title=Kristallsee&gesuggestededit=1&veaction=edit
23:30:22         cy:command ✔  contains	a, Decline
23:30:22         cy:command ✔  click	
23:30:22             cy:xhr ➟  POST http://127.0.0.1:9413/api.php
23:30:22                       Status: 200
23:30:22         cy:command ✔  get	.ve-ui-editCheckActionWidget
23:30:22         cy:command ✔  find	label
23:30:22         cy:command ✔  first	
23:30:22         cy:command ✔  click	
23:30:22         cy:command ✔  get	.ve-ui-editCheckActionWidget
23:30:22         cy:command ✘  contains	button, Submit
23:30:22         cy:command ✘  assert	expected **subject** not to be **disabled**
23:30:22                       Actual: 	"subject"
23:30:22                       Expected: 	"disabled"
23:30:22 
23:30:22 
23:30:32       ✓ Shows the Revise Tone Edit Check and tags edits (9848ms)
23:30:32     On mobile
23:30:32       - Closes the Editor when declining Edits and suggests a new task
23:30:32       - Shows the Revise Tone Edit Check and tags edits
23:30:32 
23:30:32 
23:30:32   1 passing (40s)
23:30:32   2 pending
23:30:32   1 failing
23:30:32 
23:30:32   1) Revise Tone
23:30:32        On desktop
23:30:32          Closes the Editor when declining Edits and suggests a new task:
23:30:32      AssertionError: Timed out retrying after 20000ms: Expected to find content: 'Submit' within the element: <div.oo-ui-widget.oo-ui-labelElement.oo-ui-flaggedElement-warning.oo-ui-iconElement.oo-ui-messageWidget-block.oo-ui-messageWidget.ve-ui-editCheckActionWidget.ve-ui-editCheckActionWidget-collapsed> and with the selector: 'button' but never did.
23:30:32       at Context.eval (webpack://GrowthExperiments/./cypress/e2e/ReviseTone.cy.ts:45:75)
23:30:32 
23:30:32 
23:30:32 
23:30:33 
23:30:33   (Results)
23:30:33 
23:30:33   ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
23:30:33   │ Tests:        4                                                                                │
23:30:33   │ Passing:      1                                                                                │
23:30:33   │ Failing:      1                                                                                │
23:30:33   │ Pending:      2                                                                                │
23:30:33   │ Skipped:      0                                                                                │
23:30:33   │ Screenshots:  1                                                                                │
23:30:33   │ Video:        true                                                                             │
23:30:33   │ Duration:     39 seconds                                                                       │
23:30:33   │ Spec Ran:     ReviseTone.cy.ts                                                                 │
23:30:33   └────────────────────────────────────────────────────────────────────────────────────────────────┘
23:30:33 
23:30:33 
23:30:33   (Screenshots)
23:30:33 
23:30:33   -  /workspace/log/GrowthExperiments/ReviseTone.cy.ts/Revise Tone -- On desktop -- C     (1280x720)
23:30:33      loses the Editor when declining Edits and suggests a new task (failed).png                     
23:30:33 
23:30:33 
23:30:33   (Video)
23:30:33 
23:30:33   -  Video output: /workspace/log/GrowthExperiments/ReviseTone.cy.ts.mp4
Reedy raised the priority of this task from Medium to High.Oct 21 2025, 10:59 PM

This is repeatedly blocking patches being merged to master, and in the case of one just merged, it's going to upset beta as the vendor patch has merged, but the core one hasn't.

Ugh, why do these keep failing one after another? :/

This seems closer to what this task is actually about: the Tone Check crashing before we can submit the decline-survey. Only now(?) it is also happening on desktop. See the video: https://integration.wikimedia.org/ci/job/quibble-with-gated-extensions-selenium-php81/6977/artifact/log/GrowthExperiments/ReviseTone.cy.ts.mp4

Change #1198007 had a related patch set uploaded (by Michael Große; author: Michael Große):

[mediawiki/extensions/GrowthExperiments@master] tests(ReviseTone): Skip flaky Cypress test

https://gerrit.wikimedia.org/r/1198007

Change #1196106 had a related patch set uploaded (by Michael Große; author: Michael Große):

[mediawiki/extensions/GrowthExperiments@master] DNM: run only the flaky browser tests for debugging

https://gerrit.wikimedia.org/r/1196106

Change #1198007 merged by jenkins-bot:

[mediawiki/extensions/GrowthExperiments@master] tests(ReviseTone): Skip flaky Cypress test

https://gerrit.wikimedia.org/r/1198007

Still seeing ReviseTone.cy.ts CI failures on unrelated patches -- e.g. https://integration.wikimedia.org/ci/job/quibble-with-gated-extensions-selenium-php81/8132/console

14:39:18         cy:command ✔  get	.ve-ui-toolbar-saveButton
14:39:18         cy:command ✔  assert	expected **<a.oo-ui-tool-link.ve-ui-toolbar-saveButton>** to be **visible**
14:39:18         cy:command ✔  click	
14:39:18         cy:command ✘  get	.ve-ui-mwSaveDialog .oo-ui-processDialog-actions-primary
14:39:18         cy:command ✘  assert	expected **.ve-ui-mwSaveDialog .oo-ui-processDialog-actions-primary** to be **visible**
14:39:18                       Actual: 	".ve-ui-mwSaveDialog .oo-ui-processDialog-actions-primary"
14:39:18                       Expected: 	"visible"
14:39:18           cy:fetch ➟  POST https://api.wikimedia.org/service/lw/inference/v1/models/edit-check:predict
14:39:18                       Status: 200

https://integration.wikimedia.org/ci/job/quibble-with-gated-extensions-selenium-php81/6338/console

16:17:26   1) Revise Tone
16:17:26        On mobile
16:17:26          Shows the Revise Tone Edit Check and tags edits:
16:17:26      AssertionError: Timed out retrying after 20000ms: Expected to find element: `.ve-ui-mwSaveDialog .oo-ui-processDialog-actions-primary`, but never found it.
16:17:26       at Context.eval (webpack://GrowthExperiments/./cypress/e2e/ReviseTone.cy.ts:116:72)

Same in https://integration.wikimedia.org/ci/job/quibble-with-gated-extensions-selenium-php81/8762/console (31 October):

1) Revise Tone
     On desktop
       Shows the Revise Tone Edit Check and tags edits:
   AssertionError: Timed out retrying after 20000ms: Expected to find element: `.ve-ui-mwSaveDialog .oo-ui-processDialog-actions-primary`, but never found it.
    at Context.eval (webpack://GrowthExperiments/./cypress/e2e/ReviseTone.cy.ts:65:72)

Change #1201096 had a related patch set uploaded (by Sergio Gimeno; author: Sergio Gimeno):

[mediawiki/extensions/GrowthExperiments@master] cypress(ReviseTone): skip all tests

https://gerrit.wikimedia.org/r/1201096

Change #1201096 merged by jenkins-bot:

[mediawiki/extensions/GrowthExperiments@master] cypress(ReviseTone): skip all tests

https://gerrit.wikimedia.org/r/1201096

Change #1207907 had a related patch set uploaded (by Reedy; author: Sergio Gimeno):

[mediawiki/extensions/GrowthExperiments@REL1_45] cypress(ReviseTone): skip all tests

https://gerrit.wikimedia.org/r/1207907

Change #1207907 merged by jenkins-bot:

[mediawiki/extensions/GrowthExperiments@REL1_45] cypress(ReviseTone): skip all tests

https://gerrit.wikimedia.org/r/1207907

Change #1239342 had a related patch set uploaded (by Michael Große; author: Michael Große):

[mediawiki/extensions/GrowthExperiments@master] tests(ReviseTone): reenable remaining Cypress tests

https://gerrit.wikimedia.org/r/1239342

Change #1196106 abandoned by Michael Große:

[mediawiki/extensions/GrowthExperiments@master] DNM: run only the flaky browser tests for debugging

Reason:

reenabled again in Iad51357c568081abf58798637b1938654d90819b

https://gerrit.wikimedia.org/r/1196106