Page MenuHomePhabricator

Urdu language utilities
Open, LowestPublic

Description

  • generate TFiDF badword lists
  • review and aggregation of badwords/informal words by native speaker
  • implement revscoring.Language (Language utility)

Event Timeline

ToAruShiroiNeko updated the task description. (Show Details)
ToAruShiroiNeko raised the priority of this task from to Normal.
ToAruShiroiNeko assigned this task to Ladsgroup.
ToAruShiroiNeko added a subscriber: ToAruShiroiNeko.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptDec 3 2015, 6:20 AM
Halfak set Security to None.

These files seem to have wrongly encoded, so we cant see any proper text except meaningless characters.

you need to either download them and open it with notepad (gedit, or anything suitable) or in your browser check for encoding option (probably in view menu) and choose "UTF-8" or "Unicode"

There's nothing wrong with these files regarding encoding.

Halfak added a subscriber: Halfak.Feb 24 2016, 8:12 PM

I fixed the encoding and copied it to the wiki here: https://meta.wikimedia.org/wiki/Research:Revision_scoring_as_a_service/Word_lists/ur

@MuhammadShuaib, could you take another look? We need you to move words from the "list-generated" to list-badwords(racial slurs, curse words, offensive language) and list-informals(causal talk: e.g., "hello", "haha", "lol", "wat"). Please let me know if you have any questions.

Looks like we need a new contact to help us with this work. Moving it back to the backlog until we can find someone.

Halfak lowered the priority of this task from Normal to Lowest.Jul 5 2016, 3:11 PM

I fixed the encoding and copied it to the wiki here: https://meta.wikimedia.org/wiki/Research:Revision_scoring_as_a_service/Word_lists/ur

@MuhammadShuaib, could you take another look? We need you to move words from the "list-generated" to list-badwords(racial slurs, curse words, offensive language) and list-informals(causal talk: e.g., "hello", "haha", "lol", "wat"). Please let me know if you have any questions.

Bad words have been moved under bad words.

Looks like we need a new contact to help us with this work. Moving it back to the backlog until we can find someone.

I am here @Halfak.

@MuhammadShuaib Thanks. Please tell us when you're done with informal words.

This comment was removed by Ladsgroup.
Restricted Application added a project: artificial-intelligence. · View Herald TranscriptApr 17 2019, 7:16 PM