Gather language assets for Swedish
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Halfak
	Apr 1 2016, 9:00 AM

Description

Run Bad-Words-Detection-System to get potential badword list
Human review of BWDS list
Integrate into revscoring

Details

	Subject	Repo	Branch	Lines +/-
	ores: install aspell-sv	operations/puppet	production	+1 -1

Customize query in gerrit

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Resolved		Ladsgroup	T131450 Gather language assets for Swedish
		Resolved		Ladsgroup	T135604 Train a `reverted` model for svwiki

Event Timeline

Halfak created this task.Apr 1 2016, 9:00 AM

Restricted Application added subscribers: Josve05a, Aklapper. · View Herald TranscriptApr 1 2016, 9:00 AM

@Ladsgroup, can you run Bad-Words-Detection-System on svwiki?

Halfak edited projects, added Machine-Learning-Team (Active Tasks); removed Machine-Learning-Team.Apr 3 2016, 8:25 AM

Halfak updated the task description. (Show Details)

Halfak added a project: Bad-Words-Detection-System.

Started the bot. You'll have the results in this page

Done

Is thee anything I, as a Swede, can do to help.

@Josve05a It would be great if you review the list generated by bot and divide them into three lists:

Words that are not acceptable to use anywhere in Wikipedia like 'f**ck, 'sh*t'. I understand these words are okay to use in article related to the subject but that's okay since we are considering proportion of added/changed words not total number of them
Words that are not okay in Wikipedia articles but it's okay to use them in talk pages like 'Hey', 'LOL', etc.
Words that are none of the above so false positives picked by bot.

Put it somewhere and we will do the rest.

Thanks

Ladsgroup moved this task from Parked to Backlog on the Machine-Learning-Team (Active Tasks) board.Apr 4 2016, 11:44 AM

In T131450#2177336, @Ladsgroup wrote:

@Josve05a It would be great if you review the list generated by bot and divide them into three lists:

Words that are not acceptable to use anywhere in Wikipedia like 'f**ck, 'sh*t'. I understand these words are okay to use in article related to the subject but that's okay since we are considering proportion of added/changed words not total number of them

Words that are not okay in Wikipedia articles but it's okay to use them in talk pages like 'Hey', 'LOL', etc.

Words that are none of the above so false positives picked by bot.

Put it somewhere and we will do the rest.

Thanks

Ok thanks, I'll get on that!

I've started here https://meta.wikimedia.org/wiki/Research:Revision_scoring_as_a_service/Word_lists/sv/Josve05a_sandbox

It's great! thank you. Please keep in mind that you should not sort "generated common words" only "generated words". I'm saying this since you're list contains 379 "Unknown" words but all of generated bad words are 250 words so my guess is you are also working on generated common words too.

Halfak moved this task from Backlog to Parked on the Machine-Learning-Team (Active Tasks) board.Apr 5 2016, 3:31 PM

I've gone through the rest of them, sorted them and left comments on a group of words that are often used in vandalism but I guess would have more common legitimate uses than most of the words on the list. @Josve05a , feel free to through my edits and see if there's anything you disagree with – some of the ones left weren't exactly obvious.

Ladsgroup edited projects, added Machine-Learning-Team; removed Machine-Learning-Team (Active Tasks).Apr 18 2016, 3:11 PM

Halfak updated the task description. (Show Details)May 12 2016, 8:18 PM

Halfak edited projects, added Machine-Learning-Team (Active Tasks); removed Machine-Learning-Team.

Halfak assigned this task to Ladsgroup.May 16 2016, 3:57 PM

Halfak moved this task from Parked to Backlog on the Machine-Learning-Team (Active Tasks) board.