Page MenuHomePhabricator

Add support for Uzbek (https://uz.wikipedia.org)
Open, LowestPublic

Description

To-do:

  1. Create TFiDF generated lists of potential badwords (revscoring team)
  2. Manual curation of TFiDF lists (native speaker)
  3. Train and test reverted prediction model (revscoring team)
  4. Translate edit quality campaign materials (native speaker)
  5. Deploy edit quality campaign for uzwiki (revscoring team)

.... Wikipedians label edits as "damaging" and/or "goodfaith" ... (uzwiki-pedians)

  1. Train and test damaging and goodfaith models (revscoring team)

Event Timeline

bmansurov created this task.Dec 1 2015, 2:11 AM
bmansurov updated the task description. (Show Details)
bmansurov raised the priority of this task from to Needs Triage.
bmansurov added a subscriber: bmansurov.
Restricted Application added subscribers: StudiesWorld, Aklapper. · View Herald TranscriptDec 1 2015, 2:11 AM
Krenair added a subscriber: Krenair.Dec 1 2015, 2:58 AM
Halfak updated the task description. (Show Details)Dec 4 2015, 8:09 PM
Halfak set Security to None.
Halfak updated the task description. (Show Details)

I can help with points 2 and 4.

Halfak added subscribers: Ladsgroup, Halfak.EditedDec 23 2015, 4:23 AM

@Ladsgroup, could you kick off Bad-Words-Detection-System on this?

I just started it :) we will have it by tomorrow (worst case scenario)

http://tools.wmflabs.org/dexbot/uz_badwords.txt
http://tools.wmflabs.org/dexbot/uz_stopwords.txt

It seems Uzbek Wikipedia is too small so my results may be as good as we expect.

Thanks for moving his forward, guys. What are the bad words in the above link? I see some words in that list are not bad at all.

Thanks for doing this :)
See this:
https://meta.wikimedia.org/wiki/Research_talk:Revision_scoring_as_a_service#Badwords
We should split this list to three lists: 1- swear words or words are not okay to use at all in an wikipedia. you can choose words like "shit" that are sometime okay but not most of the time 2- informal words like "hey" or "lol" that are okay to use in talk pages but not articles 3- false positives.

Anything I can do to help with the list of bad words? Should I upload a file of real bad words here?

The best thing you can do is splitting this list to three ones I mentioned earlier and please add anything you think is proper to add.

Here are the sorted lists.

Bad words: none

Informal words (or non-words, words from other languages, etc.):

#abell
#adc
#aggregator
#aliens
#alles
#alto
#amerikal
#amphibian
#and
#animal
#aquatic
#are
#bacon
#barcelona
#bhutan
#bird
#body
#bog
#borderless
#broken
#bufo
#business
#but
#cedro
#cobs
#com
#comes
#computer
#contributions
#delete
#descended
#destiny
#didn
#dies
#dil
#disney
#djobs
#dreyer
#earth
#eats
#even
#example
#face
#faisal
#familie
#file
#find
#firefox
#flag
#flower
#for
#frizzoni
#frmt
#from
#fruit
#fterran
#gangetica
#give
#godwin
#google
#googleblog
#googletoolbarinstaller
#guugal
#gwyddoniaeth
#havol
#her
#heritage
#him
#his
#homework
#hubble
#huddi
#huruf
#inc
#indir
#internet
#iphone
#ipo
#ipod
#islamabad
#its
#ittifog
#iyis
#jasminum
#jobs
#jobsning
#jpg
#just
#karakouram
#keklik
#komputerlardan
#ldi
#lgan
#lib
#lmagan
#long
#lumotlarni
#lyom
#macworld
#mail
#making
#mammal
#mandate
#marine
#markhor
#mauntin
#mavirize
#millard
#mintag
#mockbuster
#mom
#monsters
#mosque
#most
#mountain
#naaltsoos
#nci
#ndors
#net
#nuke
#obss
#officinale
#olan
#page
#pager
#palestine
#palestinearab
#palestinian
#palo
#pampered
#parco
#pedrengo
#people
#persiancrocodile
#personalized
#pha
#phe
#picasa
#pig
#pirates
#pixar
#platanista
#playeri
#playerni
#please
#power
#president
#pro
#redirect
#reptile
#resimleri
#rida
#rips
#rovlarini
#rovlarni
#sanalritim
#saudiyya
#savage
#scientia
#sea
#sellme
#silicon
#skaitlis
#snake
#source
#stephanus
#stevejobs
#stomaticus
#stupid
#symbols
#text
#the
#then
#this
#thumb
#times
#tip
#toolbar
#tried
#use
#utc
#uzbek
#valley
#veda
#viel
#view
#visionary
#vyuda
#walt
#want
#wants
#was
#whose
#wiki
#wikipedia
#windows
#yxat
#zbek
#zida
#ziga

False positives:

#agregator
#aksiyalarini
#aksiyanerga
#aksiyasiga
#animatsion
#bezi
#daromad
#dod
#dollarilik
#domeyni
#ego
#formalarni
#gugl
#gugldan
#guglning
#imzolaydilar
#indeksatsiyalagan
#kod
#kompaniya
#kompaniyasini
#kontsepsiya
#korporatsiyaning
#lentapediyada
#liq
#manbaiga
#mansablari
#maxsus
#multimediya
#multserialning
#narxlangandi
#oynalarni
#oʻsmirlik
#passport
#peyjer
#prototipini
#qidiruv
#registratsiya
#sezonga
#sezoni
#sezonning
#son
#sotuvi
#sotuvidan
#stenford
#stolida
#studiyaning
#studiyasini
#sumalak
#syujetni
#syujetning
#syurrealistik
#tabrik
#tavsif
#telefonlari
#tizimidir
#utilitilari
#voqealarni
#voy
#xuddi
#yangiliklarga
#yetarli
#yoki
#yotiq
#yumor

Just noticed that this card got stalled. It looks like we'll need another source of curses/racial slurs in Uzbek. Any ideas @bmansurov ?

Yes, you're right. I can try to find some bad words.

@bmansurov, could you take another look at this? We'd like to get uzbek supported soon. :)

Halfak updated the task description. (Show Details)Jul 15 2016, 2:38 PM
Halfak triaged this task as Low priority.
Halfak lowered the priority of this task from Low to Lowest.

@Halfak, thanks for pinging. I feel bad for linking these but here we go:

https://new.vk.com/topic-2907260_4551760
and
http://www.youswear.com/index.asp?language=Uzbek

There are many more swear words than these but I haven't been able to find a source yet. I'll keep adding more when I come across any similar reference.

Restricted Application added a project: artificial-intelligence. · View Herald TranscriptApr 17 2019, 7:19 PM