[MILESTONE] Conduct usability tests for Tone Check proof of concept
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	ppelberg
	Jan 17 2025, 12:13 AM

Description

This task involves the work of conducting a usability test of the Peacock Check proof of concept that integrates the first iteration of the peacock language detection model the ML Team is developing.

More broadly, this task supports WE 1.2.13:

If we conduct usability tests of an initial engineered version of Peacock Check with ≥10 newcomers and Junior Contributors and ≥80% of them describe the experience using terms like "helpful," "makes sense," and "clear", then we can be confident the proposed UX has the potential to lower the rate at which the new content edits are reverted on the grounds of WP:WTW (and related policies)

Decision(s) to be made

Research Questions

To what – if any – extent did people find the feedback Edit Check offered confusingly/unhelpfully generic?
- Context: this question is a response to us deciding we can be okay with flagging non-neutral language even if the type of non-neutral language varies across cases. | See Slack and comments from @jhsoby, @Strainu, and @matej_suchanek in T388215.

Related Objects
Search...

Status	Assigned	Task
Open	None	T265163 Create a system to encode best practices into editing experiences
Open	None	T365300 Introduce Edit Checks that encourage behavior aligned with Manual of Style consensus
Open	None	T365301 Tone Check: Prompt people to revise promotional language
Open	None	T416406 [MILESTONE] Offer Tone Check as default-on feature (all wikis)
Open	None	T413026 [MILESTONE] Offer Tone Check as default-on feature at partner wikis
Open	ppelberg	T392954 [FY25-26] WE 1.1: Increase constructive edits
Open	MNeisler	T387918 [MILESTONE] Run an A/B test to evaluate impact of Tone Check
Resolved	nayoub	T383956 [MILESTONE] Conduct usability tests for Tone Check proof of concept
Resolved	Esanders	T387901 Build proof of concepts for Tone Check (mobile + desktop)
Resolved	ppelberg	T383954 Define the user experience for surfacing feedback while people are typing (mobile)
Resolved	DLynch	T383955 Create proof of concept(s) for surface feedback to people while they are typing (mobile)
Resolved	gkyziridis	T386100 Create and deploy peacock detection model server
Resolved	achou	T387984 Use SHAP values to highlight peacock words
Resolved	dchan	T390248 Define Tone Check card/hint copy and calls to action
Resolved	dchan	T390250 Prototype Tone Check "Revise" workflow
Resolved	bmartinezcalvo	T390310 Tone Check: Conduct holistic review of the proof of concepts
Resolved	achou	T386645 Evaluate the existing peacock detection model
Resolved	ppelberg	T388940 Adjust Tone Check model (v1) based on internal review
Resolved	Sucheta-Salgaonkar-WMF	T384651 Evaluate efficacy of Tone Check model output (internal review)
Resolved	Quiddity	T389989 Share Tone Check proof of concept with experienced volunteers
Resolved	nayoub	T390239 Ensure article text appears "above" Edit Check highlight
Resolved	Esanders	T269145 Use CSS 'mix-blend-mode: multiply' for fake selections
Resolved	nayoub	T391143 Define the events that will cause Tone Check to be displayed
Resolved	nayoub	T391152 Define what text will be highlighted in Tone Check MVP
Resolved	MRaishWMF	T391162 [FY25-WE.1.2.13] Tone Check concept validation
Resolved	zoe	T392871 Create Tone Check Patch demo for concept validation

Event Timeline

ppelberg created this task.Jan 17 2025, 12:13 AM

ppelberg renamed this task from Conduct usability tests for how we might surface feedback to people while they are typing to Conduct usability tests for Peacock Check proof of concept.Mar 4 2025, 8:07 PM

ppelberg added a subtask: T387901: Build proof of concepts for Tone Check (mobile + desktop).

ppelberg moved this task from Untriaged to Upcoming on the Editing-team board.

ppelberg moved this task from Backlog to Triaged on the EditCheck board.

ppelberg moved this task from To Triage to Triaged on the VisualEditor board.

ppelberg mentioned this in T387901: Build proof of concepts for Tone Check (mobile + desktop).Mar 10 2025, 10:58 PM

ppelberg renamed this task from Conduct usability tests for Peacock Check proof of concept to [MILESTONE] Conduct usability tests for Peacock Check proof of concept.Mar 14 2025, 9:16 PM

ppelberg added a subtask: T384651: Evaluate efficacy of Tone Check model output (internal review).

ppelberg added a subtask: T380779: Enable Peacock model to identify positive examples with finer granularity.Mar 14 2025, 9:19 PM

ppelberg removed a parent task: T383954: Define the user experience for surfacing feedback while people are typing (mobile).Mar 14 2025, 9:21 PM

ppelberg added a parent task: T387918: [MILESTONE] Run an A/B test to evaluate impact of Tone Check.

ppelberg added a subtask: T380781: Investigate expanding language coverage for Peacock detection model.Mar 14 2025, 9:24 PM

ppelberg added a subtask: T386645: Evaluate the existing peacock detection model.Mar 14 2025, 9:29 PM

ppelberg mentioned this in T379397: Build the first iteration of Tone Check, backed by ML.

ppelberg removed a subtask: T384651: Evaluate efficacy of Tone Check model output (internal review).Mar 14 2025, 9:37 PM

ppelberg updated the task description. (Show Details)Mar 18 2025, 4:49 PM

ppelberg added subscribers: Strainu, matej_suchanek, jhsoby.

ppelberg added a project: Design-Research.Mar 25 2025, 10:32 PM

ppelberg added a subtask: T390239: Ensure article text appears "above" Edit Check highlight.Mar 28 2025, 10:11 PM

ppelberg added a subtask: T391162: [FY25-WE.1.2.13] Tone Check concept validation.Apr 16 2025, 7:28 PM

VPuffetMichel closed subtask T390239: Ensure article text appears "above" Edit Check highlight as Resolved.Apr 26 2025, 3:45 PM

DKumar-WMF changed the status of subtask T391162: [FY25-WE.1.2.13] Tone Check concept validation from Open to In Progress.May 6 2025, 1:54 PM

ppelberg removed a subtask: T380779: Enable Peacock model to identify positive examples with finer granularity.May 9 2025, 8:10 PM

ppelberg closed subtask T391162: [FY25-WE.1.2.13] Tone Check concept validation as Resolved.May 15 2025, 9:59 PM

ppelberg closed subtask T391143: Define the events that will cause Tone Check to be displayed as Resolved.May 19 2025, 5:11 PM

Aklapper renamed this task from [MILESTONE] Conduct usability tests for Peacock Check proof of concept to [MILESTONE] Conduct usability tests for Tone Check proof of concept.May 28 2025, 11:43 AM

achou closed subtask T386645: Evaluate the existing peacock detection model as Resolved.Jul 7 2025, 2:21 PM

In all testing the Editing Team completed (1, 2, 3, 4 – volunteers (new, experienced, and across geographies) described the Peacock Check experience as familiar, encouraging, and supportive.

These reactions have led the Editing Team think the Peacock Check UX is ready for deployment because we are confident newcomers will intuitively understand its purpose and be receptive to the feedback it's offering them.

Some highlights...

This would encourage me to [edit]. I said one of the reasons I didn't contribute was because I read some of the contributions and I think they're really well written and I think I can't write as well as that. My English isn't strong enough, but [with PC] I feel now that I could write something useful.

I'm for it because if somebody doesn't want it to be neutral, they know that. So they're just going to ignore it and post it. And if it's a situation where I do want to do the right thing essentially, then it's good to catch that maybe my language could be better.

I think it’s really interesting. I’m averse to the style suggestions of other programs. They’re usually kind of in bad faith—they’re bad suggestions. This is different. There’s no suggestion of what you should actually be saying...

Major lessons

People find Edit Check familiar and reminiscent of suggestions/feedback they receive on other platforms.
- Knowing this gives the Editing Team added confidence to shift our attention to populating the core UX with new Suggestions and Checks.
Readers and Editors vary in how they understand neutrality
- Experienced readers think of neutrality as being associated with a lack of emotion.
- Experienced editors refers to the degree of alignment between a source and the in-article claims it supports.
Readers remain uncertain about what caused Peacock Check to become activated.
- This finding points to the potential need for us to invest further in a UX that makes it easier for people to identify the "offending" language and the know-how needed to "neutralize" it.