Page MenuHomePhabricator
Create Task
Maniphest T407031

Revise Tone: Performance metrics are collected
Closed, ResolvedPublic3 Estimated Story Points
Actions

Description

In addition to tracking the detailed analytics needed for evaluating the A/B test (T405177), we also want engineering focused performance metrics to be able to assert that our feature works as expected on a technical level.

This explicitly includes tracking the runtime needed to figure out for which paragraph to show the Revise Tone Check. (See the internal slack conversation for context.)

Acceptance Criteria:
Metrics for at least the following are being recorded:

  • time to get data from Data Gateway
  • time to get find correct paragraph in article
  • number of users declining a task with which reason in the survey
  • number of errors
    • suggestion no longer available (no data in data gateway)
    • paragraph ambiguous (distance of 2nd-highest scoring paragraph is 90% of the distance of the 2nd-highest scoring paragraph)

In addition to the above, we should make sure that the ReviseTone task is visible on the KPI dashboard next to the other Newcomer Tasks

  • ReviseTone will be visible on the KPI dashboard

Notes:

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
fix(ReviseTone): use correct stats metrics namemediawiki/extensions/GrowthExperimentsmaster+1 -1
stats(ReviseTone): count clicks to retake the onboarding quizmediawiki/extensions/GrowthExperimentsmaster+8 -0
stats(ReviseTone): Record language for quiz responsesmediawiki/extensions/GrowthExperimentsmaster+5 -0
metrics(ReviseTone): track distance of paragraph match and ambiguitymediawiki/extensions/GrowthExperimentsmaster+39 -1
metrics(ReviseTone): record platform for Levenshtein performancemediawiki/extensions/GrowthExperimentsmaster+1 -0
metrics(ReviseTone): track quiz responsesmediawiki/extensions/GrowthExperimentsmaster+23 -1
metrics(ReviseTone): record the time to match the paragraphmediawiki/extensions/GrowthExperimentsmaster+9 -0
Customize query in gerrit

Related Objects

Event Timeline

Michael created this task.Oct 10 2025, 7:24 PM
Restricted Application added a project: Growth-Team. · View Herald TranscriptOct 10 2025, 7:24 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Michael mentioned this in T407356: Create Production Revise Tone Recommendation Provider.Oct 21 2025, 11:01 AM
KStoller-WMF triaged this task as High priority.Oct 22 2025, 3:44 PM
KStoller-WMF moved this task from Inbox to Needs Estimation on the Growth-Team board.
Michael added a project: OKR-Work.Oct 27 2025, 3:20 PM
Michael updated the task description. (Show Details)Oct 30 2025, 4:26 PM
Michael renamed this task from Revise Tone: Performance metrics are collected and visualized to Revise Tone: Performance metrics are collected.Nov 3 2025, 5:01 PM
Michael updated the task description. (Show Details)
KStoller-WMF moved this task from Needs Estimation to Estimated tasks backlog on the Growth-Team board.Nov 4 2025, 4:57 PM
KStoller-WMF lowered the priority of this task from High to Medium.Nov 4 2025, 4:59 PM
• Iflorez subscribed.Nov 4 2025, 5:00 PM
Michael updated the task description. (Show Details)Nov 12 2025, 4:56 PM
Michael moved this task from Estimated tasks backlog to Needs Estimation on the Growth-Team board.Nov 17 2025, 4:58 PM
Michael updated the task description. (Show Details)Nov 26 2025, 5:40 PM
KStoller-WMF subscribed.Dec 1 2025, 5:21 PM
Michael updated the task description. (Show Details)Dec 1 2025, 5:22 PM
KStoller-WMF set the point value for this task to 3.Dec 1 2025, 5:22 PM
KStoller-WMF moved this task from Needs Estimation to Estimated tasks backlog on the Growth-Team board.
KStoller-WMF moved this task from Estimated tasks backlog to FY2025-26 Q2 Sprint 5 on the Growth-Team board.Dec 2 2025, 6:02 PM
KStoller-WMF edited projects, added Growth-Team (FY2025-26 Q2 Sprint 5); removed Growth-Team.
Michael claimed this task.Dec 4 2025, 3:24 PM
Restricted Application added a project: User-Michael. · View Herald TranscriptDec 4 2025, 3:24 PM
Michael moved this task from Incoming to Doing on the Growth-Team (FY2025-26 Q2 Sprint 5) board.Dec 4 2025, 6:44 PM
Michael mentioned this in T383563: mw.track: support for histogram metrics.Dec 4 2025, 7:34 PM
gerritbot added a comment.Dec 4 2025, 7:41 PM

Change #1215246 had a related patch set uploaded (by Michael Große; author: Michael Große):

[mediawiki/extensions/GrowthExperiments@master] metrics(ReviseTone): record the time to match the paragraph

https://gerrit.wikimedia.org/r/1215246

gerritbot added a project: Patch-For-Review.Dec 4 2025, 7:41 PM
Michael updated the task description. (Show Details)Dec 5 2025, 2:00 PM

I extended the definition of the task type variable on the KPI dashboard to include revise_tone.

gerritbot added a comment.Dec 8 2025, 9:44 AM

Change #1215246 merged by jenkins-bot:

[mediawiki/extensions/GrowthExperiments@master] metrics(ReviseTone): record the time to match the paragraph

https://gerrit.wikimedia.org/r/1215246

ReleaseTaggerBot added a project: MW-1.46-notes (1.46.0-wmf.7; 2025-12-16).Dec 8 2025, 10:00 AM
Maintenance_bot removed a project: Patch-For-Review.Dec 8 2025, 10:30 AM
Michael updated the task description. (Show Details)Dec 9 2025, 2:04 PM
gerritbot added a comment.Dec 11 2025, 11:22 AM

Change #1217486 had a related patch set uploaded (by Michael Große; author: Michael Große):

[mediawiki/extensions/GrowthExperiments@master] metrics(ReviseTone): track distance of paragraph match

https://gerrit.wikimedia.org/r/1217486

gerritbot added a project: Patch-For-Review.Dec 11 2025, 11:22 AM
Michael moved this task from Doing to Code Review on the Growth-Team (FY2025-26 Q2 Sprint 5) board.Dec 11 2025, 5:11 PM
Urbanecm_WMF edited projects, added Growth-Team (FY2025-26 Q2 Sprint 6); removed Growth-Team (FY2025-26 Q2 Sprint 5).Dec 16 2025, 5:39 PM
Urbanecm_WMF moved this task from Incoming to Code Review on the Growth-Team (FY2025-26 Q2 Sprint 6) board.
gerritbot added a comment.Dec 17 2025, 4:53 PM

Change #1219183 had a related patch set uploaded (by Michael Große; author: Michael Große):

[mediawiki/extensions/GrowthExperiments@master] metrics(ReviseTone): record platform for Levenshtein performance

https://gerrit.wikimedia.org/r/1219183

gerritbot added a comment.Dec 18 2025, 11:33 AM

Change #1219176 had a related patch set uploaded (by Michael Große; author: Michael Große):

[mediawiki/extensions/GrowthExperiments@master] metrics(ReviseTone): track quiz responses

https://gerrit.wikimedia.org/r/1219176

gerritbot added a comment.Dec 19 2025, 3:44 PM

Change #1219176 merged by jenkins-bot:

[mediawiki/extensions/GrowthExperiments@master] metrics(ReviseTone): track quiz responses

https://gerrit.wikimedia.org/r/1219176

gerritbot added a comment.Dec 22 2025, 9:05 AM

Change #1219183 merged by jenkins-bot:

[mediawiki/extensions/GrowthExperiments@master] metrics(ReviseTone): record platform for Levenshtein performance

https://gerrit.wikimedia.org/r/1219183

KStoller-WMF edited projects, added Growth-Team (FY2025-26 Q3 Sprint 1); removed Growth-Team (FY2025-26 Q2 Sprint 6).Jan 13 2026, 7:53 PM
KStoller-WMF moved this task from Incoming to Code Review on the Growth-Team (FY2025-26 Q3 Sprint 1) board.
gerritbot added a comment.Jan 26 2026, 6:32 PM

Change #1233240 had a related patch set uploaded (by Michael Große; author: Michael Große):

[mediawiki/extensions/GrowthExperiments@master] stats(ReviseTone): count clicks to retake the onboarding quiz

https://gerrit.wikimedia.org/r/1233240

gerritbot added a comment.Jan 26 2026, 6:57 PM

Change #1233248 had a related patch set uploaded (by Michael Große; author: Michael Große):

[mediawiki/extensions/GrowthExperiments@master] stats(ReviseTone): Record language for quiz responses

https://gerrit.wikimedia.org/r/1233248

KStoller-WMF added a subscriber: Dyolf77_WMF.Jan 27 2026, 4:56 PM

@Michael - We reviewed the Revise Tone dashboard in our weekly Ambassador meeting, and @Dyolf77_WMF asked if it's possible to filter by wiki.

That would be ideal, especially since there is a chance that translations impact quiz understanding. Is that an easy addition, impossible, or possible but enough work that I should create a new task to support this request.

Also, we still haven't seen any new arwiki edits, so it would be good to understand if we have any quiz clicks on arwiki.

Michael edited projects, added Growth-Team (FY2025-26 Q3 Sprint 2); removed Growth-Team (FY2025-26 Q3 Sprint 1).Jan 27 2026, 5:47 PM
Michael moved this task from Incoming to Code Review on the Growth-Team (FY2025-26 Q3 Sprint 2) board.
Michael added a comment.Jan 27 2026, 7:21 PM
In T407031#11558699, @KStoller-WMF wrote:

@Michael - We reviewed the Revise Tone dashboard in our weekly Ambassador meeting, and @Dyolf77_WMF asked if it's possible to filter by wiki.

That would be ideal, especially since there is a chance that translations impact quiz understanding. Is that an easy addition, impossible, or possible but enough work that I should create a new task to support this request.

Also, we still haven't seen any new arwiki edits, so it would be good to understand if we have any quiz clicks on arwiki.

One of the changes that I have added to this task yesterday is about collecting exactly that data! Once it has been merged and deployed (which might still be a while), we can look into adjusting the dashboard to see what is going on.

gerritbot added a comment.Jan 29 2026, 3:50 PM

Change #1217486 merged by jenkins-bot:

[mediawiki/extensions/GrowthExperiments@master] metrics(ReviseTone): track distance of paragraph match and ambiguity

https://gerrit.wikimedia.org/r/1217486

ReleaseTaggerBot edited projects, added MW-1.46-notes (1.46.0-wmf.14; 2026-02-03); removed MW-1.46-notes (1.46.0-wmf.7; 2025-12-16).Jan 29 2026, 4:00 PM
gerritbot added a comment.Jan 29 2026, 4:20 PM

Change #1233240 merged by jenkins-bot:

[mediawiki/extensions/GrowthExperiments@master] stats(ReviseTone): count clicks to retake the onboarding quiz

https://gerrit.wikimedia.org/r/1233240

gerritbot added a comment.Jan 29 2026, 4:27 PM

Change #1233248 merged by jenkins-bot:

[mediawiki/extensions/GrowthExperiments@master] stats(ReviseTone): Record language for quiz responses

https://gerrit.wikimedia.org/r/1233248

Maintenance_bot removed a project: Patch-For-Review.Jan 29 2026, 4:32 PM
gerritbot added a comment.Jan 29 2026, 7:18 PM

Change #1235097 had a related patch set uploaded (by Michael Große; author: Michael Große):

[mediawiki/extensions/GrowthExperiments@master] fix(ReviseTone): use correct stats metrics name

https://gerrit.wikimedia.org/r/1235097

gerritbot added a project: Patch-For-Review.Jan 29 2026, 7:18 PM
gerritbot added a comment.Jan 30 2026, 11:58 AM

Change #1235097 merged by jenkins-bot:

[mediawiki/extensions/GrowthExperiments@master] fix(ReviseTone): use correct stats metrics name

https://gerrit.wikimedia.org/r/1235097

Maintenance_bot removed a project: Patch-For-Review.Jan 30 2026, 12:44 PM
Michael moved this task from Code Review to Doing on the Growth-Team (FY2025-26 Q3 Sprint 2) board.Feb 10 2026, 5:35 PM
Michael edited projects, added Growth-Team (FY2025-26 Q3 Sprint 3); removed Growth-Team (FY2025-26 Q3 Sprint 2).Feb 10 2026, 6:01 PM
Michael moved this task from Incoming to Doing on the Growth-Team (FY2025-26 Q3 Sprint 3) board.
Michael updated the task description. (Show Details)Feb 10 2026, 7:43 PM
Michael moved this task from Doing to Code Review on the Growth-Team (FY2025-26 Q3 Sprint 3) board.

I think this dashboard now contains all the data that we have in Prometheus for Revise Tone: https://grafana.wikimedia.org/d/0352e22b-1ebf-4e13-a97d-4f5f05eb8463/revise-tone-structured-task

Please have a look and let me know if anything is unclear or could be done better!

KStoller-WMF added a comment.Feb 17 2026, 1:36 AM

I'm so glad we have easy access to the quiz data, thank you for prepping this dashboard!

"Number of clicks to retake the onboarding quiz on platform" <--- Does that refer to when a user opens the help panel and clicks into the onboarding quiz again?

Michael added a comment.Feb 17 2026, 12:33 PM
In T407031#11621373, @KStoller-WMF wrote:

I'm so glad we have easy access to the quiz data, thank you for prepping this dashboard!

"Number of clicks to retake the onboarding quiz on platform" <--- Does that refer to when a user opens the help panel and clicks into the onboarding quiz again?

Yes, exactly! I'll add a note to it.

Michael closed this task as Resolved.Feb 24 2026, 5:36 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits