Page MenuHomePhabricator

Verify cost of gathering peacock training/evaluation data for top 20 languages
Closed, ResolvedPublic

Description

We want to verify the effort/cost of collecting training and evaluation data for the top 20 languages.

The top 20 languages are English (en), French (fr), German (de), Spanish (es), Japanese (ja), Russian (ru), Portuguese (pt), Italian (it), Chinese (zh), Persian (fa), Polish (pl), Arabic (ar), Dutch (nl), Ukrainian (uk), Romanian (ro), Hebrew (he), Indonesian (id), Norwegian (no), Turkish (tr), Czech (cs).

In T386645, we use two approaches to collect fine-grained paragraph/sentence-level labels for peacock behavior data for English:

  1. finding reverted edits that mention [[WP:PEACOCK]] (a link to the Wikipedia: Manual of Style#Puffery page) in their reverting comments, then extracting the changed paragraphs from these edits.
  2. finding sentences marked with template:Peacock_inline.

To verify whether these approaches work for non-English languages and assess the available data, we need to check:

  1. if each language has its own Wikipedia: Manual of Style#Puffery page, and what are the abbreviations/shortcuts (e.g., "WP:PEACOCK"), and if communities actively use these abbreviations when reverting edits containing peacock language.
  2. if template:Peacock_inline exists in each language and what name is used for it.

Related Objects

Event Timeline

achou changed the task status from Open to In Progress.Mar 13 2025, 12:05 PM
achou updated the task description. (Show Details)

On English Wikipedia, there are two places that discuss peacock language:

  1. https://en.wikipedia.org/wiki/Wikipedia:Manual_of_Style/Words_to_watch#Puffery: This is the "Puffery" section under "Words that may introduce bias". It is part of Wikipedia's policies or guidelines, and the shortcut [[WP:PEACOCK]] redirects here.

The page exists in the 19 non-English targeted languages. However, not all languages have a dedicated section about peacock language or an associated shortcut. We'll need to investigate each language further individually.

  1. https://en.wikipedia.org/wiki/Wikipedia:Wikipuffery: This is a dedicated essay about peacock language.

The page exists in 10 non-English targeted languages (ar, es, fa, fr, id, ja, nl, pl, ro, zh), and 9 of these have shortcuts linked to the page, as follows:

{
    'fr': ["WP:ELS", "WP:TROP"],
    'es': ["WP:EEL", "WP:EJV"],
    'ja': ["WP:APT", "WP:PEACOCK", "WP:TAIGEN"],
    'zh': ["WP:APT", "WP:PEACOCK", "WP:華而不實"],
    'fa': ["وپ:اکتاک","وپ:تکلف","وپ:تمجید"],
    'pl': ["WP:UPW","WP:Peacock","WP:Pustosłowie","WP:Mowa-trawa","WP:PAW"],
    'ar': ["وب:تباه"],
    'ro': ["WP:EJV"],
    'id': ["WP:PUFF","WP:HIPERBOLIS","WP:PEACOCK"]
}

I used these abbreviations/shortcuts to search for reverted edits in these languages. The following are the number of edits we found:

fr 26                                                                                
es 259                                                                                
ja 301
zh 18                                                                                
fa 23                                                                                
pl 57                                                                            
ar 66                                                                                
ro 71
id 15

This limited data may be because communities don't commonly mention these specific abbreviations/shortcuts when reverting edits that contains peacock language, instead using other keywords we haven't identified.

The page exists in the 19 non-English targeted languages. However, not all languages have a dedicated section about peacock language or an associated shortcut. We'll need to investigate each language further individually.

FWIW: For the Norwegian Bokmål Wikipedia (nowiki), we don't have a dedicated page or section about this (and therefore no abbreviation either, naturally) that is distinct from the more general page (whose title is roughly equivalent to "Wikipedia:Avoid weasel words").

In Romanian we have:

  • WP:EJV - peacock words, value judgments
  • WP:FE - weasel words (peocock are included here)

The two are used more or less interchangably, I would check for both, although some will not be strictly PEACOCK reverts.

In Czech Wikipedia, I think there is no commonly understood term for "peacock words." It simply falls under NPOV policy.

We do discourage weasel words (WP:VSVS), and there is a handful of inline templates for requiring clarification. And we also have some edit filters tagging edits with suspicious phrases (and even warning the users), such as "undoubtedly", "luckily", "unfortunately", etc., but also language like "last century", "until recently", "in our country", etc. Not sure if this is within the scope.

Update on the second approach:

The template "Peacock_inline" exists in five non-English targeted languages (ar, fa, id, pt, zh), with the following names:

{
    'ar': ["لفظ إطراء" ,"تباهي" ,"تباه"],
    'fa': ["بزرگنمایی"],
    'id': ["Berlebihan", "Peacock term"],
    'pt': ["Conteúdo parcial"],
    'zh': ["华而不实", "華而不實"],
}

Using the same method, we searched these templates in mediawiki_wikitext_history and found the following revision matches:

ar 1774                                                                                
fa 326                                                                         
id 22                                                                           
pt 11608
zh 52

Note that these revisions contain identical paragraphs/sentences that appear across different revisions of the same page, so a further cleanup step is needed.

Based on our initial analysis of data for the top 19 non-English languages:

  • Languages that have enough eval data: Spanish (es), Japanese (ja), Arabic (ar), Portuguese (pt).
  • Languages that have "Peacock" guidelines or templates but have limited data:
    • French (fr), Romanian (ro), Polish (pl), Persian (fa), Chinese (zh), Indonesian (id)
  • Languages that don't have "Peacock" guidelines or templates:
    • German (de), Russian (ru), Italian (it), Dutch (nl), Ukrainian (uk), Hebrew (he), Turkish (tr), Norwegian (no), Czech (cs)

Ideas for gathering more data:

  • Expand search criteria to include more general guidelines like "Terms to use with caution," "Elusive content," or "Weasel words". Based on feedback from @jhsoby, @Strainu, and @matej_suchanek (Thanks a lot!!), it appears that some languages don't make strong distinctions between peacock words, weasel words, and NPOV issues.
  • Consult with the community to identify common terms and templates used when flagging or reverting these types of edits.

Based on feedback from @jhsoby, @Strainu, and @matej_suchanek (Thanks a lot!!), it appears that some languages don't make strong distinctions between peacock words, weasel words, and NPOV issues.

+1, @achou! Thank you for all of this wonderful input, @jhsoby, @Strainu, and @matej_suchanek ^ _ ^

Related: when the model is ready for review, might you all be interested in helping us to review it? If so, could you please indicate as much by commenting what language(s) you could review the model within on T388471?

And hey, zero expectations about the above! Y'all have already been tremendously helpful.

Resolved this task, as we have the following follow-up tasks:
T398930: Score probability evaluation for languages without enough data.
T389445: Compile list of templates, jargon and policies relevant to NPOV.