Next step

With @MNeisler having defined the Requirements, this task is now ready to be worked on.

Arguably this is redundant with the tag in T388716? Insofar as if that tag is applied and the user is in the control group (as seen via the bucket on other events), we'd know that the user would have been shown a tone check. We wouldn't have the specific information about whether they would have been shown it midedit or presave.

Arguably this is redundant with the tag in T388716?

Yes, the tag to be created in T388716 will help identify published edits in the control group that would have been shown tone check. This is all that is needed for the majority of our metrics, including the KPIs (revert rate and proportion of edits without biased language) that look specifically at published edits.

The current gap in instrumentation is identifying edit attempts that would have been shown tone check but were not published. This primarily impacts the Edit completion rate/Edit abandonment guardrails metrics if we define as proportion of edits completed after the moment tone check was or could have been shown.

As an alternative, we could look at overall edit completion rate changes from moments already defined in the editing workflow such as from saveIntent. The main problem is that, for the control group, this would include non-content type edits such as small typo fixes, which might inherently have a different edit completion rate than content-type edits.

I'm glad you raised this @DLynch and thank you @MNeisler for naming this nuanced difference.

...primarily impacts the Edit completion rate/Edit abandonment guardrails metrics if we define as proportion of edits completed after the moment tone check was or could have been shown.

Per what Megan and I discussed offline (and Megan noted above), we will need to be able to track Edit completion rate/Edit abandonment guardrails metrics so that we can evaluate changes in edit quality Tone Check might cause alongside changes it might cause in the likelihood that people proceed to publish an edit.

With the above in mind, we'd like to move forward with adding the instrumentation this task is asking for.

Change #1152384 had a related patch set uploaded (by DLynch; author: DLynch):

[mediawiki/extensions/VisualEditor@master] Edit check: log to VEFU if a tone check was filtered before being shown

https://gerrit.wikimedia.org/r/1152384

Can someone confirm that this tag will not be shown publicly (in diffs, in history pages, RecentChanges, etc.)? I'd like to avoid the confusion we observed with RefCheck, when some tags were understood as "it was shown to the user" while it was the tag for the control group.

In T394952#10947328, @Trizek-WMF wrote:

Can someone confirm that this tag will not be shown publicly (in diffs, in history pages, RecentChanges, etc.)? I'd like to avoid the confusion we observed with RefCheck, when some tags were understood as "it was shown to the user" while it was the tag for the control group.

@Trizek-WMF: good question and yes, these events will NOT be shown on-wiki.

@DLynch: Megan and I are wondering about the same question that we asked in T393818#10948614: would it be accurate for us to think the instrumentation specified here is deployed and available for testing via ecenable=2?

No, the patch isn't merged yet.

In T394952#10948639, @DLynch wrote:

No, the patch isn't merged yet.

Got it. Thank you for confirming. What – if anything – is blocking it from being merged?

Mostly that we haven't actually defined/implemented the a/b test bucketing yet, so "log when someone is in the control group" is somewhat meaningless.

Documenting what Megan, David, and I discussed offline (in a distinct conversations): this work will be ready for QA once the A/B test begins.

I updated this patch on top of the bucketing implementation I did for T389231. I had to rewrite it to cope with how we're handling the "filtering" with the async check.

Because tone can be shown during an edit, this is going to be be able to trigger more frequently than the equivalent for add-reference did. Specifically: once a tone violation exists in the document, it's going to be triggering this event every time the user moves between paragraphs. (And then once in the pre-save.)

Change #1184121 had a related patch set uploaded (by DLynch; author: DLynch):

[mediawiki/extensions/VisualEditor@wmf/1.45.0-wmf.16] Edit check: log to VEFU if a tone check would have been shown if not for the a/b test

https://gerrit.wikimedia.org/r/1184121

Change #1152384 merged by jenkins-bot:

[mediawiki/extensions/VisualEditor@master] Edit check: log to VEFU if a tone check would have been shown if not for the a/b test

https://gerrit.wikimedia.org/r/1152384

Change #1184124 had a related patch set uploaded (by DLynch; author: DLynch):

[mediawiki/extensions/VisualEditor@wmf/1.45.0-wmf.17] Edit check: log to VEFU if a tone check would have been shown if not for the a/b test

https://gerrit.wikimedia.org/r/1184124

Change #1184121 merged by jenkins-bot:

[mediawiki/extensions/VisualEditor@wmf/1.45.0-wmf.16] Edit check: log to VEFU if a tone check would have been shown if not for the a/b test

https://gerrit.wikimedia.org/r/1184121

Mentioned in SAL (#wikimedia-operations) [2025-09-02T20:18:36Z] <kemayo@deploy1003> Started scap sync-world: Backport for [[gerrit:1184115|Edit check: set up the tone check a/b test (T389231 T402195)]], [[gerrit:1184121|Edit check: log to VEFU if a tone check would have been shown if not for the a/b test (T394952)]]

Mentioned in SAL (#wikimedia-operations) [2025-09-02T20:24:49Z] <kemayo@deploy1003> kemayo: Backport for [[gerrit:1184115|Edit check: set up the tone check a/b test (T389231 T402195)]], [[gerrit:1184121|Edit check: log to VEFU if a tone check would have been shown if not for the a/b test (T394952)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.

Mentioned in SAL (#wikimedia-operations) [2025-09-02T20:31:11Z] <kemayo@deploy1003> Finished scap sync-world: Backport for [[gerrit:1184115|Edit check: set up the tone check a/b test (T389231 T402195)]], [[gerrit:1184121|Edit check: log to VEFU if a tone check would have been shown if not for the a/b test (T394952)]] (duration: 12m 34s)

Change #1184124 merged by jenkins-bot:

[mediawiki/extensions/VisualEditor@wmf/1.45.0-wmf.17] Edit check: log to VEFU if a tone check would have been shown if not for the a/b test

https://gerrit.wikimedia.org/r/1184124

Mentioned in SAL (#wikimedia-operations) [2025-09-03T20:34:20Z] <kemayo@deploy1003> Started scap sync-world: Backport for [[gerrit:1184124|Edit check: log to VEFU if a tone check would have been shown if not for the a/b test (T394952)]], [[gerrit:1184120|Edit check: deploy tone a/b test to frwiki, jawiki, ptwiki (T389231)]]

Mentioned in SAL (#wikimedia-operations) [2025-09-03T20:38:58Z] <kemayo@deploy1003> kemayo: Backport for [[gerrit:1184124|Edit check: log to VEFU if a tone check would have been shown if not for the a/b test (T394952)]], [[gerrit:1184120|Edit check: deploy tone a/b test to frwiki, jawiki, ptwiki (T389231)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.

Mentioned in SAL (#wikimedia-operations) [2025-09-03T20:45:32Z] <kemayo@deploy1003> Finished scap sync-world: Backport for [[gerrit:1184124|Edit check: log to VEFU if a tone check would have been shown if not for the a/b test (T394952)]], [[gerrit:1184120|Edit check: deploy tone a/b test to frwiki, jawiki, ptwiki (T389231)]] (duration: 11m 12s)

I've confirmed that this event is logged correctly in VEFU following deployment of the Tone Check AB Test. See summary of current events logged to date below:

test_group	                action	                       feature	       n_sessions  n_users
2025-09-editcheck-tone-control	check-control-branchNodeChange	editCheck-tone  42	   40

Note: This reflects the number of sessions logged since 8 September 2025 once the fix for T403745 was implemented.

The number of distinct editing sessions where this event is logged aligns with the frequency of check-shown-presave and check-shown-midedit events in the test group.