Invite volunteers to review Peacock language model (v1)
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	ppelberg
	Mar 10 2025, 11:03 PM

Description

In T384651, we are conducting at an initial internal review of the Peacock language detection model.

This task builds on this initial review by inviting volunteers to evaluate the evaluations the model has made so that we we can identify what – if any – adjustments needs to made to it in order to meet the precision thresholds we've set.

Stories

As an experienced volunteer/moderator motivated to ensure that new(er) volunteers are receiving helpful feedback that aligns with Wikipedia policies, I want to inspect the evaluations the Peacock Check model is making so that I can be confident the feedback new(er) volunteers will ultimately be receiving is constructive for them and the wiki at-large.
As an experienced volunteer/moderator understandably skeptical of AI...
As a tester, I need a simple interface to check the model.

Volunteer Review Requirements

In order for WMF staff to evaluate the performance of the model using volunteer input, they will need to:

Recruit at least 3 (but ideally 5+) human evaluators per language
Use fresh content in our eval dataset (published in last 12 months)

Previously, we'd listed the requirements of Annotool here. Although, we've since moved them to T392324.

Looking for volunteers

If you want to help us to review of the Peacock language detection model, please signup here.
The list of languages for this first phase is final. Please avoid adding another language.
The first phase of evaluation will start with the languages listed as Priority == yes and Eval Data Status == Available.
This is a first test. The end goal remains to provide Peacock check to all wikis.

Languages

Wiki	Language	Priority – `Editing Team`	Eval Data Status – `ML Team`	Human Evaluation Status	Volunteer Contacts – `Editing Team`	Notes
ar.wiki	Arabic	Yes	Available	Annotool not yet ready	signup for test	This community is potentially interested in testing the model.
cs.wiki	Czech	Yes	TBD	Annotool not yet ready	@matej_suchanek
de.wiki	German		TBD	Annotool not yet ready
en.wiki	English	Yes	Available	⏳ Annotool set up in progress	signup for test
es.wiki	Spanish	Yes	Available	⏳ Annotool set up in progress	signup for test	This community is interested in testing the model.
fa.wiki	Persian		TBD	Annotool not yet ready	@Huji, @Ladsgroup
fr.wiki	French	Yes	TBD	⏳ Annotool set up in progress
he.wiki	Hebrew	Yes	TBD	Annotool not yet ready
hy.wiki	Armenian		Not available	Annotool not yet ready	@Mari_Avetisyan_WMAM
id.wiki	Indonesian	Yes	TBD	Annotool not yet ready		This community is interested in testing the model.
it.wiki	Italian		TBD	Annotool not yet ready
ja.wiki	Japanese	Yes	Available	⏳ Annotool set up in progress	signup for test
mk.wiki	Macedonian		Not available	Annotool not yet ready	@Ehrlich91
nl.wiki	Dutch		TBD	Annotool not yet ready
no.wiki	Norwegian Bokmål		TBD	Annotool not yet ready
pl.wiki	Polish	Yes	TBD	Annotool not yet ready	@PMG, @Msz2001, Sławek Borewicz
pt.wiki	Portuguese	Yes	Available	⏳ Annotool set up in progress	signup for test
ro.wiki	Romanian		TBD	Annotool not yet ready	@Strainu
ru.wiki	Russian		TBD	Annotool not yet ready	@Iniquity
tr.wiki	Turkish	Yes	TBD	Annotool not yet ready
uk.wiki	Ukrainian		TBD	Annotool not yet ready
uz.wiki	Uzbek		TBD	Annotool not yet ready	@Panpanchik
zh.wiki	Chinese	Yes	TBD	Annotool not yet ready	@SCP-2000, @Hamishcn, @Yiming, @Stang

NOTE: languages currently noted as "Editing Team Priorities" reflect: 1) technical priorities per T388471#10781906, 2) projects that see relatively high volumes of newcomers, specifically newcomers living in Sub-Saharan Africa, and 3) projects that have expressed a willingness to experiment with Peacock Check

Process

✅ ~~Define what tool we'll user to gather feedback through~~
✅ ~~Define requirements for evaluation data~~ *
T392324: Prepare annotool for Tone Check model evaluation
Invite volunteer input
Evaluate results

*NOTE: if we choose to move forward with Annotool, the instance might need to be updated to meet evaluation requirement (e.g. free form text for reviewers to express nuance around evaluation)

Open questions

1. What tool/format will we invite volunteers to offer feedback using? E.g. might we build a dedicated UI (see "References" below)? Might we use a spreadsheet? Something else?
Per T388471#10755841, we're going to use Annotool to gather input. The work to prepare Annotool will happen in T392324.
2. How many edits will we need each language community to review for the results to be meaningful to the ML Team?

References

mw:Add an image/2025 algorithm test via Growth Team
https://annotool.toolforge.org/projects/8
https://alis-evaluation.toolforge.org/
- A tool the Growth Team built to evaluate article-level image suggestions
https://editcheck.sthottingal.workers.dev/?revision=1263744496
- A tool @santhosh wrote that leverages an LLM to detect policy violations within edits. T382631 is a follow-up task that would augment this tool to enable a crowd of people to evaluate the model's performance.

Related Objects
Search...

Status	Assigned	Task
Open	None	T265163 Create a system to encode best practices into editing experiences
Open	None	T365300 Introduce Edit Checks that encourage behavior aligned with Manual of Style consensus
Open	None	T365301 Tone Check: Prompt people to revise promotional language
Open	ppelberg	T392954 [FY25-26] WE 1.1: Increase constructive edits
Open	Quiddity	T409513 Publish Tone Check leading indicator findings on mediawiki.org
Open	MNeisler	T387918 [MILESTONE] Run an A/B test to evaluate impact of Tone Check
Resolved	Trizek-WMF	T388471 Invite volunteers to review Peacock language model (v1)
Resolved	ppelberg	T387925 Determine language support for Tone Check (v1)
Resolved	achou	T388215 Verify cost of gathering peacock training/evaluation data for top 20 languages
Open	None	T388736 [SPIKE] Create a lightweight estimate for proportion of new content edits reverted because of peacock language
Resolved	Trizek-WMF	T389445 Compile list of templates, jargon and policies relevant to NPOV
Resolved	ppelberg	T388937 Create way for volunteers to review and offer feedback about Tone Check model (v1)

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

In T388471#10655083, @ppelberg wrote:

If/when we come to learn volunteers do not find the spreadsheet approach – which Sucheta is going to mockup – easeful enough, we'll consider alternative approaches.

Oh, and @jhsoby, since you're here: might you be open to reviewing/offering feedback about the spreadsheet Sucheta is going to draft?

Hi. Responding to T388215#10655076, I can help review the model in Romanian.

Czech (cs.wiki)

In T388471#10655725, @ppelberg wrote:

In T388471#10655083, @ppelberg wrote:

If/when we come to learn volunteers do not find the spreadsheet approach – which Sucheta is going to mockup – easeful enough, we'll consider alternative approaches.

Oh, and @jhobsy, since you're here: might you be open to reviewing/offering feedback about the spreadsheet Sucheta is going to draft?

Sure 👍

Yiming subscribed.Mar 21 2025, 6:45 AM

@jhsoby, @Matej_Orlicky, and @Strainu: wonderful! Thank you for being open to this, y'all 🙏🏼

Next step
@SSalgaonkar-WMF will be sharing two things for you to review:

The mock spreadsheet I referred to in T388471#10655725
A prototype for what it could look to evaluate the model using a dedicated tool (https://annotool.toolforge.org/)

ppelberg assigned this task to Sucheta-Salgaonkar-WMF.Mar 26 2025, 11:48 PM

ppelberg moved this task from Backlog to Dependent on others on the Editing-team (Tracking) board.

Hello, I can help review the model in Chinese (zh.wiki). Thanks.

In T388471#10688240, @SCP-2000 wrote:

Hello, I can help review the model in Chinese (zh.wiki). Thanks.

Oh, wonderful! Thank you, @SCP-2000. We'll @ mention you here when there is something ready for you to review...

ppelberg updated the task description. (Show Details)Mar 31 2025, 11:49 PM

ppelberg updated the task description. (Show Details)Mar 31 2025, 11:53 PM

Johannnes89 subscribed.Apr 1 2025, 1:44 PM

ppelberg updated the task description. (Show Details)Apr 2 2025, 6:59 PM

In T388471#10681234, @ppelberg wrote:

...two things for you to review:

The mock spreadsheet I referred to in T388471#10655725

A prototype for what it could look to evaluate the model using a dedicated tool (https://annotool.toolforge.org/)

Update

Alright! The two artifacts are ready for review...

Request
@jhsoby, @Matej_Orlicky, @SCP-2000, @Strainu, and anyone else here: Might you be able to review the two artifacts below (spreadsheet + annotool)?

We're specifically interested in learning:

Which of the two options below do you think would be most easeful to use to offer feedback about the model's evaluations?
What – if any – information beyond the below can you see yourself needing in order to express an opinion about whether you think an edit introduces non-neutral language?
- Text an edit adds/changes
- Paragraph the added/modified text is a part of
- Name of article the edit was made to
- Diff showing the paragraph before and after the change

Format	Option #1: Spreadsheet	Option #2: Annotool
Screenshot
Link to review	EXAMPLE: Peacock language detection - model evaluation	https://annotool.toolforge.org/projects/11

WARNING: for demonstration purposes, both options above are only showing edits in English. When we're ready for y'all to review actual edits, evaluation data will be made available for each of the languages listed in the task description.

NOTE: a big thank you to @MunizaA for creating the annotool instance 🙏🏼👏🏼

ppelberg claimed this task.Apr 4 2025, 8:03 PM

ppelberg edited projects, added Editing-team (Kanban Board); removed Editing-team (Tracking).

ppelberg moved this task from Inbox to Doing on the Editing-team (Kanban Board) board.

ppelberg updated the task description. (Show Details)Apr 4 2025, 8:08 PM

Huji updated the task description. (Show Details)Apr 4 2025, 10:22 PM

I added myself to help with Persian. I also think the wrong Matej was tagged by @ppelberg above; @matej_suchanek is the on who should have been tagged.

Personally, I prefer option #2 and embedding the review process into the Wiki interface.

@Ladsgroup this may be of interest to you too. And the wiki-embedded tooling you created for your revert-bot (Shahbaz) may be helpful here too.

Per Huji, I would prefer option 2. Perhaps we can write a remark for each individual edit if available, instead of simply tagging "yes" or "no". Thank you.

According to what was stated above, I can help review the model in Arabic (arwiki)

In my opinion, opt. 2 is the most easeful. The labeling narrows to simple yes/no, and it also contains the diff information (which the check will also use). But we won't get the "dictionary" (If yes, what words or phrases should be detected?).

In T388471#10714647, @Huji wrote:

I added myself to help with Persian. I also think the wrong Matej was tagged by @ppelberg above; @matej_suchanek is the on who should have been tagged.

Personally, I prefer option #2 and embedding the review process into the Wiki interface.

I agree, I dream a day, we could add such information on diff page and feed back these data to the ML infra for re-train of the models or online learning.

@Ladsgroup this may be of interest to you too. And the wiki-embedded tooling you created for your revert-bot (Shahbaz) may be helpful here too.

Yeah. I think I can be both ways. Keep me in the loop.

Trizek-WMF mentioned this in T389445: Compile list of templates, jargon and policies relevant to NPOV.Apr 16 2025, 3:26 PM

BAPerdana-WMF subscribed.Apr 16 2025, 3:30 PM

Sucheta-Salgaonkar-WMF updated the task description. (Show Details)Apr 17 2025, 11:19 PM

Sucheta-Salgaonkar-WMF updated the task description. (Show Details)Apr 17 2025, 11:44 PM

ppelberg updated the task description. (Show Details)Apr 18 2025, 11:00 PM

Update

In T388471#10714315, @ppelberg wrote:

We're specifically interested in learning:

Which of the two options below do you think would be most easeful to use to offer feedback about the model's evaluations?

What – if any – information beyond the below can you see yourself needing in order to express an opinion about whether you think an edit introduces non-neutral language?

@Huji, @Ladsgroup, @matej_suchanek, @SCP-2000: thank you all for thinking through this decision with us and sharing what you think about them. 🙏🏼

DECIDED
We're going to move forward with the approach y'all converged on: Option #2: Annotool.

NEXT STEPS

The ML and Editing Teams will be working together to...
1. Update annotool with: A) a text field wherein y'all (volunteers) can offer context about what specifically you consider to be neutral about a given diff/edit and B) evaluation data for the languages listed in the task description
  - Note: this work will happen in T392324.
As evaluation data becomes ready ("B)"), we'll @-mention in a comment here to signal to you that you're clear to begin reviewing

While the above is happening, an ask of y'all: can you please invite a few peers you know who you think would be interested and equipped to help out with evaluating the model in the languages listed above? Ideally, there are ≥3 people/language reviewing the model.

Ok! That's it for this comment. I'm going to follow up in a separate one with responses to some other ideas/thoughts that were raised...

ppelberg updated the task description. (Show Details)Apr 18 2025, 11:17 PM

ppelberg mentioned this in T392324: Prepare annotool for Tone Check model evaluation .Apr 18 2025, 11:27 PM

ppelberg updated the task description. (Show Details)Apr 18 2025, 11:33 PM

Iniquity updated the task description. (Show Details)Apr 19 2025, 5:57 PM

Iniquity subscribed.

alaa updated the task description. (Show Details)Apr 24 2025, 10:43 PM

alaa added a subscriber: Michel_Bakni.

ppelberg updated the task description. (Show Details)Apr 25 2025, 7:59 PM

ppelberg added subscribers: Panpanchik, Mari_Avetisyan_WMAM, Ehrlich91, PMG.

@Panpanchik, @Mari_Avetisyan_WMAM, @Ehrlich91, and @PMG: I've boldly added you all as people who would be open to reviewing the Peacock Check model in Uzbek, Armenian, Macedonian, and Polish when data becomes available.

If I've at all misinterpreted your willingness to contribute to the project in this way, please do let me know (here or email!

Regardless, thank you all for all of the insightful and thoughtful comments you shared during Wednesday's CEE - APP Call 🙏🏼

Hi, if it's okay for you, I can help w/ zhwiki's.

Sdkb subscribed.Apr 28 2025, 7:09 PM

ppelberg mentioned this in T393051: Tone Check: Offer people more clarity about which part of the text is non-neutral.Apr 30 2025, 7:31 PM

In T388471#10770002, @Hamishcn wrote:

Hi, if it's okay for you, I can help w/ zhwiki's.

You helping with zh.wiki would be wonderful, @Hamishcn would be wonderful! Thank you for volunteering ^ _ ^

A resulting question: might there be a page on zh.wiki that people are likely to check who would be interested in also reviewing the model?

Reason I ask: we're seeking 3 volunteers/wiki./

ppelberg updated the task description. (Show Details)Apr 30 2025, 7:42 PM

Language priorities

While the languages we prioritize supporting to start will depend, in large part, on what evaluation data is available (Sucheta and Aiko are looking into this), the Editing Teams deems the following languages as priorities:

Wikis that use the “variants” feature (Chinese) — because the model has to infer across different language varieties
Languages that don’t space-separate words (Chinese, Japanese) — where the results will be very dependent on the tokenizer
Agglutinative languages (Turkish, Indonesian) — where the model will be very dependent on the tokenizer
RTL (Arabic, Hebrew) — because of UX issues

I've updated the task description to reflect the above...

ppelberg updated the task description. (Show Details)Apr 30 2025, 7:54 PM

Hi @ppelberg, I can help zhwiki. :)

Could give a hand on zhwiki if still needed

In T388471#10781902, @ppelberg wrote:

In T388471#10770002, @Hamishcn wrote:

Hi, if it's okay for you, I can help w/ zhwiki's.

You helping with zh.wiki would be wonderful, @Hamishcn would be wonderful! Thank you for volunteering ^ _ ^

A resulting question: might there be a page on zh.wiki that people are likely to check who would be interested in also reviewing the model?

Reason I ask: we're seeking 3 volunteers/wiki./

Appreciate, looks like no need of posts on zhwiki now.:)

In T388471#10781939, @Yiming wrote:

Hi @ppelberg, I can help zhwiki. :)

! In T388471#10782832, @Stang wrote:

Could give a hand on zhwiki if still needed

@Yiming + @Stang: Amazing! We'd value both of y'alls help – thank you for offering ^ _ ^

I'll @ mention you when there is something ready for you to see...

ppelberg updated the task description. (Show Details)May 1 2025, 7:28 PM

Sucheta-Salgaonkar-WMF updated the task description. (Show Details)May 1 2025, 9:20 PM

ppelberg moved this task from Doing to Blocked / Needs More Work on the Editing-team (Kanban Board) board.May 6 2025, 5:15 PM

ppelberg updated the task description. (Show Details)May 6 2025, 7:23 PM

ppelberg updated the task description. (Show Details)May 6 2025, 7:26 PM

I can help with Polish

VPuffetMichel added a project: OKR-Work.May 8 2025, 1:54 PM

In T388471#10799496, @Msz2001 wrote:

I can help with Polish

Wonderful! Thank you, @Msz2001 – I'll @ mention you when the evaluation data is ready.

ppelberg updated the task description. (Show Details)May 8 2025, 4:09 PM

I found another volunteer for pl.wiki:
https://pl.wikipedia.org/wiki/Wikipedysta:S%C5%82awek_Borewicz

ppelberg updated the task description. (Show Details)May 9 2025, 7:48 PM

In T388471#10806011, @PMG wrote:

I found another volunteer for pl.wiki:
https://pl.wikipedia.org/wiki/Wikipedysta:S%C5%82awek_Borewicz

Wonderful, @PMG – thank you! I've updated the task description to include them: T388471#10814490.

achou mentioned this in T391940: FY2024-25 Q4 Goal: Productionize tone check model.May 13 2025, 1:12 PM

Trizek-WMF updated the task description. (Show Details)May 13 2025, 1:26 PM

Trizek-WMF updated the task description. (Show Details)May 14 2025, 4:00 PM

Michael subscribed.May 15 2025, 3:21 PM

Trizek-WMF updated the task description. (Show Details)May 15 2025, 4:22 PM

ppelberg mentioned this in T394448: Expand language coverage for Tone Check.May 15 2025, 6:18 PM

ppelberg added a project: Goal.May 15 2025, 7:21 PM

Jilucorg subscribed.May 17 2025, 5:32 PM

ppelberg updated the task description. (Show Details)May 19 2025, 8:05 PM

ppelberg updated the task description. (Show Details)

Jilucorg unsubscribed.May 19 2025, 9:21 PM

ppelberg reassigned this task from ppelberg to Trizek-WMF.May 20 2025, 5:21 PM

ppelberg mentioned this in T389231: Deploy config change to start the Tone Check A/B Test.May 27 2025, 11:16 PM

Chipmunkdavis subscribed.Jun 4 2025, 4:36 AM

The first phase of evaluation is now done. The machine learning team has received enough evaluations for Spanish, Japanese, English, Portuguese, and French.

Future evaluations will be conducted and covered by a separate ticket. Users are invited to signup on wiki: https://www.mediawiki.org/wiki/Edit_check/Tone_Check/Model_evaluation

Shizhao moved this task from Research to Closed on the Chinese-Sites board.Jul 4 2025, 3:03 AM

Hi y'all! Just wanted to send a BIG thank you to everyone who participated in evaluating the Tone Check model! We really appreciate your help.

I also wanted to post a link to the results of this evaluation: https://www.mediawiki.org/wiki/Edit_check/Tone_Check#Evaluating_the_model

If you have any questions about our learnings, please use the Tone Check discussion page to get in touch: https://www.mediawiki.org/wiki/Talk:Edit_check/Tone_Check

ppelberg closed subtask T387925: Determine language support for Tone Check (v1) as Resolved.Jul 11 2025, 11:28 PM

ppelberg mentioned this in T387925: Determine language support for Tone Check (v1).

ppelberg closed subtask T388937: Create way for volunteers to review and offer feedback about Tone Check model (v1) as Resolved.Jul 11 2025, 11:32 PM

ppelberg removed a subtask: T392324: Prepare annotool for Tone Check model evaluation .Jul 11 2025, 11:36 PM

Trizek-WMF mentioned this in T403102: Invite volunteers to review Tone language model (v2).Aug 27 2025, 5:46 PM

Stang unsubscribed.Oct 14 2025, 1:27 PM