Enable Peacock model to identify positive examples with finer granularity
Open, Needs TriagePublic
Actions

Assigned To

None

Authored By

	ppelberg
	Nov 25 2024, 5:43 PM

Project Tags

Referenced Files

None

Subscribers

Description

In T368274, we developed a model capable of detecting Peacock behavior at the article level.

This task involves the work of iterating upon this approach to detect peacock behavior at the level of paragraphs, and ideally, sentences.

In doing so, we'll gain the ability to offer people Edit Checks that are relevant to the specific pieces of content they added/affected during a given edit session.

Story

As someone adding new information to Wikipedia and/or changing the meaning of information that's already present, I want to be made aware when the specific additions and/or changes I'm making are at risk of introducing non-neutral language to Wikipedia so that I can increase the likelihood that these edits remain published and avoid making destructive changes that will impact readers and create more work for moderators.

Requirements

@ppelberg to discuss with @diego.

Related Objects
Search...

Status	Assigned	Task
Open	None	T265163 Create a system to encode best practices into editing experiences
Open	None	T365300 Introduce Edit Checks that encourage behavior aligned with Manual of Style consensus
Open	None	T365301 Tone Check: Prompt people to revise promotional language
Open	None	T416406 [MILESTONE] Offer Tone Check as default-on feature (all wikis)
Resolved	Esanders	T413026 [MILESTONE] Offer Tone Check as default-on feature at partner wikis
Open	ppelberg	T392954 [FY25-26] WE 1.1: Increase constructive edits
Resolved	MNeisler	T387918 [MILESTONE] Run an A/B test to evaluate impact of Tone Check
Open	None	T380779 Enable Peacock model to identify positive examples with finer granularity
Resolved	Aitolkyn	T380569 Baseline Experiments for SDS 1.2.1 B

Event Timeline

ppelberg created this task.Nov 25 2024, 5:43 PM

The main challenge here is to find data to train and test this model. Currently, the data we have is at article label. I see to possible ways to work around this problem:

Using section level templates (eg. {{Peacock|section|date=November 2024}} ): Doing this we well be able to reduce the granularity from article to section level. To do this, we require time to collect data and run new experiments.

Trying a unsupervised approach: The idea here would be to take articles with tagged as peackock and try to find common patterns across them. This would be basically an extension of T371158#10202597, and would require research work but also some type of community or expert validation (per language).

Ideally we should try both approaches because they are complementary. The first one would allow a robust evaluation methodology, although the granularity would be just at section level. The second one, can give more insights/feedback for editors, but it is more difficult to evaluate and would require some type of user testing, implying more resources and time.

Isaac subscribed.Dec 3 2024, 1:52 PM

ppelberg added a subtask: T380569: Baseline Experiments for SDS 1.2.1 B.Dec 17 2024, 12:38 AM

isarantopoulos subscribed.Jan 8 2025, 9:28 AM

leila closed subtask T380569: Baseline Experiments for SDS 1.2.1 B as Resolved.Jan 13 2025, 3:57 PM

ppelberg edited parent tasks, added: T383956: [MILESTONE] Conduct usability tests for Tone Check proof of concept; removed: T379397: Build the first iteration of Tone Check, backed by ML.Mar 14 2025, 9:19 PM

ppelberg edited parent tasks, added: T387918: [MILESTONE] Run an A/B test to evaluate impact of Tone Check; removed: T383956: [MILESTONE] Conduct usability tests for Tone Check proof of concept.May 9 2025, 8:10 PM

@dchan: are we aligned in thinking this task can be resolved considering we've figured out a way for Tone Check to identify issues at the level of sentences?

ppelberg assigned this task to dchan.Jul 30 2025, 7:15 PM

ppelberg moved this task from Inbox to Blocked / Needs More Work on the Editing-team (Kanban Board) board.

VPuffetMichel added a project: OKR-Work.Aug 22 2025, 6:31 PM

Per today's team meeting, we've not yet evaluated the reliability of either a) the model's ability to output tone issues at a sentence level or b) VE's ability to present sentence-level model output in a reliable UX.

The above, combined with the UX choice to descope feedback of this sort from the MVP, is prompting us to remove this from

@ppelberg is this something you still want us to explore?

Enable Peacock model to identify positive examples with finer granularityOpen, Needs TriagePublicActions