Maniphest T411914

[Config] Deploy config change to STOP the Tone Check A/B experiment
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	ppelberg
	Dec 5 2025, 11:25 PM

Project Tags

Referenced Files

None

Subscribers

View All 10 Subscribers

Description

In T389231, we started the Tone Check A/B experiment. In this task, we will stop the Tone Check A/B experiment.

Timing

28 January 2026, per T411914#11501020

Requirements

Turn off the Tone Check experiment we enabled in T389231
- See T411914#11474888 for more.

Done

@MNeisler specifies the earliest date on which the Tone Check A/B experiment can be disabled
Editing Engineering deploys config change to stop the Tone Check A/B experiment
@Quiddity informs participating communities that experiment has stopped
- See more in Tone check A/B test communication plan

Details

Related Changes in Gerrit:

	Subject	Author	Repo	Branch	Lines +/-
	Edit check: turn off the tone a/b test on frwiki, jawiki, ptwiki	DLynch	operations/mediawiki-config	master	+0 -4

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Open	None	T265163 Create a system to encode best practices into editing experiences
Open	None	T365300 Introduce Edit Checks that encourage behavior aligned with Manual of Style consensus
Open	None	T365301 Tone Check: Prompt people to revise promotional language
Open	None	T416406 [MILESTONE] Offer Tone Check as default-on feature (all wikis)
Resolved	Esanders	T413026 [MILESTONE] Offer Tone Check as default-on feature at partner wikis
Open	ppelberg	T392954 [FY25-26] WE 1.1: Increase constructive edits
Resolved	MNeisler	T387918 [MILESTONE] Run an A/B test to evaluate impact of Tone Check
Resolved	DLynch	T411914 [Config] Deploy config change to STOP the Tone Check A/B experiment

Event Timeline

ppelberg created this task.Dec 5 2025, 11:25 PM

Will we be turning tone check off for everyone (on those wikis) or turning it on for everyone (on those wikis)?

@MNeisler when do we plan to stop this experiment? I'm asking as we only have a few days left before the end-of-year break. :)

Trizek-WMF mentioned this in T387918: [MILESTONE] Run an A/B test to evaluate impact of Tone Check.Dec 18 2025, 11:34 AM

Thanks for checking on this @Trizek-WMF.

I've pulled data for the analysis as we are aiming to complete an initial report by the end of this quarter; however, additional data might be helpful to increase confidence in some of the identified changes.

@ppelberg - Can you confirm next steps here? Is it ok to extend this experiment into the new year or should we go ahead and end based on current results?

In T411914#11474223, @MNeisler wrote:

Thanks for checking on this @Trizek-WMF.

I've pulled data for the analysis as we are aiming to complete an initial report by the end of this quarter; however, additional data might be helpful to increase confidence in some of the identified changes.

@ppelberg - Can you confirm next step\s here? Is it ok to extend this experiment into the new year or should we go ahead and end based on current results?

+1 – great question, Benoît.

Yes, let's leave the experiment running so that, as Megan helpfully put it, we can be confident that additional data is not needed to draw statistically significant conclusions.

In T411914#11437882, @DLynch wrote:

Will we be turning tone check off for everyone (on those wikis) or turning it on for everyone (on those wikis)?

Great spot: we'll be turning off the experiment.

The work of enabling Tone Check for newcomers will happen in T413026 and is blocked on the findings in T387918.

ppelberg updated the task description. (Show Details)Dec 19 2025, 12:20 AM

In T411914#11474881, @ppelberg wrote:

In T411914#11474223, @MNeisler wrote:

Thanks for checking on this @Trizek-WMF.

I've pulled data for the analysis as we are aiming to complete an initial report by the end of this quarter; however, additional data might be helpful to increase confidence in some of the identified changes.

@ppelberg - Can you confirm next step\s here? Is it ok to extend this experiment into the new year or should we go ahead and end based on current results?

+1 – great question, Benoît.

Yes, let's leave the experiment running so that, as Megan helpfully put it, we can be confident that additional data is not needed to draw statistically significant conclusions.

Hi all! Since the KR under which this work sits has been extended to 28 Feb, there's a bit of additional float in the schedule, if it would make enough of a statistically meaningful difference that would give us more confidence in our decision-making. Do we have a new end date in mind?

Do we have a new end date in mind?

@ldelench_wmf Thanks for the update! Based on the tone check AB test data gathered so far, I estimate that we'll need the test to run for ideally about 3 more weeks to give us sufficient statistical power needed to confirm the overall and desktop decreases we are observing in reverts.

This would put the new end date at 28 January.

ppelberg edited projects, added: Editing-team (Editing-19Jan-30Jan-2026); removed: Editing-team (Scheduled).Jan 16 2026, 8:30 PM

ppelberg updated the task description. (Show Details)

ppelberg moved this task from Decisions to be made to Ready to Be Worked On on the Editing-team (Editing-19Jan-30Jan-2026) board.

ppelberg added a subscriber: Quiddity.

ppelberg removed MNeisler as the assignee of this task.Jan 22 2026, 12:50 AM

Change #1235392 had a related patch set uploaded (by DLynch; author: DLynch):

[operations/mediawiki-config@master] Edit check: turn off the tone a/b test on frwiki, jawiki, ptwiki

https://gerrit.wikimedia.org/r/1235392

Restricted Application added a subscriber: Dragoniez. · View Herald TranscriptJan 30 2026, 8:22 PM

gerritbot added a project: Patch-For-Review.Jan 30 2026, 8:22 PM

^ that patch will undeploy it, but the next deployment window won't be until Monday

ppelberg moved this task from Ready to Be Worked On to Code Review on the Editing-team (Editing-19Jan-30Jan-2026) board.Jan 31 2026, 12:16 AM

Change #1235392 merged by jenkins-bot:

[operations/mediawiki-config@master] Edit check: turn off the tone a/b test on frwiki, jawiki, ptwiki

https://gerrit.wikimedia.org/r/1235392

Mentioned in SAL (#wikimedia-operations) [2026-02-02T21:05:24Z] <kemayo@deploy2002> Started scap sync-world: Backport for [[gerrit:1235392|Edit check: turn off the tone a/b test on frwiki, jawiki, ptwiki (T411914)]], [[gerrit:1235111|Enable suggestions BetaFeature on beta wikis (T415504)]], [[gerrit:1230462|WikimediaCustomizations: Set WMCBadEmailDomainsFile (T397244)]], [[gerrit:1235491|filebackend: Clean up removed config params for multi-write backends (T328872)]]

Mentioned in SAL (#wikimedia-operations) [2026-02-02T21:07:21Z] <kemayo@deploy2002> tgr, func, kemayo, esanders: Backport for [[gerrit:1235392|Edit check: turn off the tone a/b test on frwiki, jawiki, ptwiki (T411914)]], [[gerrit:1235111|Enable suggestions BetaFeature on beta wikis (T415504)]], [[gerrit:1230462|WikimediaCustomizations: Set WMCBadEmailDomainsFile (T397244)]], [[gerrit:1235491|filebackend: Clean up removed config params for multi-write backends (T328872)]] synced to

Mentioned in SAL (#wikimedia-operations) [2026-02-02T21:16:18Z] <kemayo@deploy2002> Finished scap sync-world: Backport for [[gerrit:1235392|Edit check: turn off the tone a/b test on frwiki, jawiki, ptwiki (T411914)]], [[gerrit:1235111|Enable suggestions BetaFeature on beta wikis (T415504)]], [[gerrit:1230462|WikimediaCustomizations: Set WMCBadEmailDomainsFile (T397244)]], [[gerrit:1235491|filebackend: Clean up removed config params for multi-write backends (T328872)]] (duration: 10