Deploy config change to start the Tone Check A/B Test
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	ppelberg
	Mar 18 2025, 4:54 PM

Description

Deployment timing

Deployment on September 3rd.

Bucketing criteria

Bucketing Requirements:

Bucketing should include both registered and unregistered users at all identified participating wikis.
All users, who are editing a desktop or mobile main namespace page (NS:0), at any of the participating wikis should have a 50% chance of being included/bucketed into the A/B test's control or treatment group.
Bucketing should be done on a per-Wikipedia basis. 50% of people within a given wiki should be placed within the control group; 50% should be bucketed in the treatment group
The test group should have Tone Check experience enabled while the control group should not
People should remain in the same test group for the duration of the test (and across sessions and pages).

NOTE: We need to specify the experiences that will be available in both the test and the control groups. For example, will all wikis in the control group have access to Multi-Check (References)?

Instrumentation-Related Requirements

A bucket is applied to these events so we can distinguish all events logged for the control group and the test group within the A/B test.
- The bucket should be descriptive of the test in case there are overlapping AB tests.
The anonymous_user_token field is populated for unregistered users in the test.

Participating wikis

Wiki	Status	Notes
~~es.wiki~~		Spanish Wikipedia should NOT be included in the A/B test that is scheduled to begin on 28 August 2025
fr.wiki		The wiki has been informed of the upcoming test.
ja.wiki		The wiki has been informed of the upcoming test.
pt.wiki		The wiki has been informed of the upcoming test.

Edit Check Configuration

#TODO

Details

Related Changes in Gerrit:

Subject	Repo	Branch	Lines +/-
Edit check: deploy tone a/b test to frwiki, jawiki, ptwiki	operations/mediawiki-config	master	+4 -0
Edit check: set up the tone check a/b test	mediawiki/extensions/VisualEditor	wmf/1.45.0-wmf.16	+25 -7
Edit check: set up the tone check a/b test	mediawiki/extensions/VisualEditor	master	+25 -7

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Open	None	T265163 Create a system to encode best practices into editing experiences
Open	None	T365300 Introduce Edit Checks that encourage behavior aligned with Manual of Style consensus
Open	None	T365301 Tone Check: Prompt people to revise promotional language
Open	ppelberg	T392954 [FY25-26] WE 1.1: Increase constructive edits
Open	Quiddity	T409513 Publish Tone Check leading indicator findings on mediawiki.org
Open	MNeisler	T387918 [MILESTONE] Run an A/B test to evaluate impact of Tone Check
Resolved	DLynch	T394952 Log edits when Tone Check would've been shown had someone not been in control group
Resolved	DLynch	T389231 Deploy config change to start the Tone Check A/B Test

Event Timeline

ppelberg created this task.Mar 18 2025, 4:54 PM

ppelberg renamed this task from Deploy config change to start the Peaocck Check A/B Test to Deploy config change to start the Peacock Check A/B Test.May 9 2025, 10:17 PM

ppelberg merged a task: T393816: Deploy config change to start the Tone Check A/B Test.

ppelberg updated the task description. (Show Details)May 27 2025, 11:16 PM

Aklapper renamed this task from Deploy config change to start the Peacock Check A/B Test to Deploy config change to start the Tone Check A/B Test.May 28 2025, 11:43 AM

ppelberg updated the task description. (Show Details)May 28 2025, 8:38 PM

MNeisler updated the task description. (Show Details)May 28 2025, 8:42 PM

MNeisler updated the task description. (Show Details)

ppelberg added a subscriber: Esanders.Jun 3 2025, 5:22 PM

ppelberg mentioned this in T395154: [Tone Check] Announce proposed start of A/B Test.Jun 4 2025, 10:42 PM

ppelberg edited projects, added Editing-team (Kanban Board); removed Editing-team (Tracking).Jun 10 2025, 9:18 PM

ppelberg added a parent task: T394952: Log edits when Tone Check would've been shown had someone not been in control group.Jun 30 2025, 5:35 PM

Michael subscribed.Jul 14 2025, 10:45 AM

Change #1180215 had a related patch set uploaded (by DLynch; author: DLynch):

[mediawiki/extensions/VisualEditor@master] Edit check: set up the tone check a/b test

https://gerrit.wikimedia.org/r/1180215

gerritbot added a project: Patch-For-Review.Aug 19 2025, 8:25 PM

DLynch mentioned this in T394952: Log edits when Tone Check would've been shown had someone not been in control group.Aug 19 2025, 9:41 PM

ppelberg mentioned this in T387918: [MILESTONE] Run an A/B test to evaluate impact of Tone Check.Aug 21 2025, 11:21 PM

Per today's team meeting, we're going to assume that @DLynch and @medelius will handle this together.

Please note: es.wiki will NOT be participating in the A/B test that is scheduled to begin tomorrow, 28 August 2025

Change #1180215 merged by jenkins-bot:

[mediawiki/extensions/VisualEditor@master] Edit check: set up the tone check a/b test

https://gerrit.wikimedia.org/r/1180215

Maintenance_bot removed a project: Patch-For-Review.Sep 1 2025, 11:31 AM

Trizek-WMF updated the task description. (Show Details)Sep 2 2025, 12:47 PM

Change #1184115 had a related patch set uploaded (by DLynch; author: DLynch):

[mediawiki/extensions/VisualEditor@wmf/1.45.0-wmf.16] Edit check: set up the tone check a/b test

https://gerrit.wikimedia.org/r/1184115

gerritbot added a project: Patch-For-Review.Sep 2 2025, 3:53 PM

Change #1184120 had a related patch set uploaded (by DLynch; author: DLynch):

[operations/mediawiki-config@master] Edit check: deploy tone a/b test to frwiki, jawiki, ptwiki

https://gerrit.wikimedia.org/r/1184120

Change #1184115 merged by jenkins-bot:

[mediawiki/extensions/VisualEditor@wmf/1.45.0-wmf.16] Edit check: set up the tone check a/b test

https://gerrit.wikimedia.org/r/1184115

Mentioned in SAL (#wikimedia-operations) [2025-09-02T20:18:36Z] <kemayo@deploy1003> Started scap sync-world: Backport for [[gerrit:1184115|Edit check: set up the tone check a/b test (T389231 T402195)]], [[gerrit:1184121|Edit check: log to VEFU if a tone check would have been shown if not for the a/b test (T394952)]]

Stashbot mentioned this in T402195: Improve edit check a/b test configuration to cope with multiple tests running side by side.Sep 2 2025, 8:18 PM

Mentioned in SAL (#wikimedia-operations) [2025-09-02T20:24:49Z] <kemayo@deploy1003> kemayo: Backport for [[gerrit:1184115|Edit check: set up the tone check a/b test (T389231 T402195)]], [[gerrit:1184121|Edit check: log to VEFU if a tone check would have been shown if not for the a/b test (T394952)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.

Mentioned in SAL (#wikimedia-operations) [2025-09-02T20:31:11Z] <kemayo@deploy1003> Finished scap sync-world: Backport for [[gerrit:1184115|Edit check: set up the tone check a/b test (T389231 T402195)]], [[gerrit:1184121|Edit check: log to VEFU if a tone check would have been shown if not for the a/b test (T394952)]] (duration: 12m 34s)

Change #1184120 merged by jenkins-bot:

[operations/mediawiki-config@master] Edit check: deploy tone a/b test to frwiki, jawiki, ptwiki

https://gerrit.wikimedia.org/r/1184120

Maintenance_bot removed a project: Patch-For-Review.Sep 3 2025, 8:31 PM

Mentioned in SAL (#wikimedia-operations) [2025-09-03T20:34:20Z] <kemayo@deploy1003> Started scap sync-world: Backport for [[gerrit:1184124|Edit check: log to VEFU if a tone check would have been shown if not for the a/b test (T394952)]], [[gerrit:1184120|Edit check: deploy tone a/b test to frwiki, jawiki, ptwiki (T389231)]]

Mentioned in SAL (#wikimedia-operations) [2025-09-03T20:38:58Z] <kemayo@deploy1003> kemayo: Backport for [[gerrit:1184124|Edit check: log to VEFU if a tone check would have been shown if not for the a/b test (T394952)]], [[gerrit:1184120|Edit check: deploy tone a/b test to frwiki, jawiki, ptwiki (T389231)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.

Mentioned in SAL (#wikimedia-operations) [2025-09-03T20:45:32Z] <kemayo@deploy1003> Finished scap sync-world: Backport for [[gerrit:1184124|Edit check: log to VEFU if a tone check would have been shown if not for the a/b test (T394952)]], [[gerrit:1184120|Edit check: deploy tone a/b test to frwiki, jawiki, ptwiki (T389231)]] (duration: 11m 12s)