Page MenuHomePhabricator
Create Task
Maniphest T121545

Wikipedia-Text–Based Language Models for Language Identification
Closed, ResolvedPublic
Actions

Description

Though T118287 shows that query-based text models work better than wiki-text–based language models, getting good query data can be hard. So, we should see if Wikipedia-based language models for languages with crappy training data do better. (could obviate the need for T121544 or T121546 in some cases; could be tested via re-test of data in T121539 or T121541 with current best language identification module)

Need list of target languages to work on (based on T121539).

Estimate: 2-4 hours per language to create corpuse, do minimal cleanup, and build & test model(s).

Related Objects
Search...

Event Timeline

TJones created this task.Dec 15 2015, 5:51 PM
TJones raised the priority of this task from to Needs Triage.
TJones updated the task description. (Show Details)
TJones added a project: CirrusSearch.
TJones added a subscriber: TJones.
Restricted Application added a project: Discovery-ARCHIVED. · View Herald TranscriptDec 15 2015, 5:51 PM
Restricted Application added subscribers: StudiesWorld, Aklapper. · View Herald Transcript
TJones mentioned this in T121544: Create Manually "Curated" Training Sets for Top N Languages for Language Identification.Dec 15 2015, 5:53 PM
TJones added a parent task: T121544: Create Manually "Curated" Training Sets for Top N Languages for Language Identification.
TJones updated the task description. (Show Details)
TJones set Security to None.
TJones added subtasks: T121539: Create Balanced Language Identification Evaluation Set for Top N Wikis by Query Volume, T121541: Create Properly Weighted Language Identification Evaluation Sets for Top N Other Wikis.
TJones mentioned this in T121546: Experiment with Equalizing Training Set Sizes for Language Identification.
TJones added a parent task: T121546: Experiment with Equalizing Training Set Sizes for Language Identification.
TJones added a parent task: T118278: [EPIC] Improve Language Identification for use in Cirrus Search.Dec 15 2015, 5:56 PM
Deskana moved this task from Inbox to Multilingual and cross-project on the CirrusSearch board.Dec 31 2015, 12:28 AM
TJones mentioned this in T118278: [EPIC] Improve Language Identification for use in Cirrus Search.Jan 6 2016, 11:37 PM
TJones mentioned this in T123537: Generate wikitext-based and query-based language models for TextCat.Jan 26 2016, 2:15 PM
Deskana triaged this task as Medium priority.Jan 27 2016, 11:09 PM
Deskana moved this task from Needs triage to Search on the Discovery-ARCHIVED board.
Deskana added a subscriber: Deskana.
TJones added a comment.Feb 1 2016, 9:30 PM

I've generated these models based on lightly cleaned up wiki-text and submitted them for review.

The models haven't been vetted, but T121539 isn't going to cover all 70 of these languages anyway.

Nemo_bis added a subscriber: Nemo_bis.Feb 3 2016, 7:25 AM
Deskana closed subtask T121539: Create Balanced Language Identification Evaluation Set for Top N Wikis by Query Volume as Resolved.May 11 2016, 10:40 PM
Restricted Application added a project: Discovery-Search. · View Herald TranscriptMay 11 2016, 10:40 PM
Deskana moved this task from needs triage to Up Next on the Discovery-Search board.May 12 2016, 10:06 PM
debt moved this task from Up Next to search-icebox on the Discovery-Search board.Jun 2 2016, 10:06 PM
TJones mentioned this in T140300: Provide language identification to the long-tail of wikis.Jul 13 2016, 8:26 PM
debt added a subscriber: debt.Aug 4 2016, 7:00 PM

From a conversation with @TJones:

This is a good place to spend time. Wikitext models are generally easy to create by pulling articles from the relevant Wikipedia for training data. The bang-for-your-buck is probably high enough to do this for all of the medium-effort section of the long-tail (many are already done)

TJones claimed this task.EditedAug 4 2016, 8:43 PM

As we are focusing on the top 25 languages by volume from the dashboard, we have either query-based models or wiki-text–based models for all 25. (Only Finnish, Hungarian, Norwegian, and Tagalog don't have query-based models.)

BTW, the top 25 are: English, German, Spanish, Portuguese, Russian, French, Italian, Japanese, Polish, Arabic, Chinese, Dutch, Turkish, Swedish, Persian, Czech, Vietnamese, Indonesian, Korean, Finnish, Hebrew, Tagalog, Thai, Norwegian, and Hungarian.)

If we need to create new wiki-text–based models while working on the current batch of language wikipedias—say, because there's a new language that's used a lot on one of these wikis—I can spin up a new subtask to build that wiki-text model.

So, I say mark this one done.

debt edited projects, added Discovery-Search (Current work); removed Discovery-Search.Aug 4 2016, 10:55 PM

Moving to done column on sprint board, based on @TJones's comment above!

debt moved this task from Incoming to Needs Reporting on the Discovery-Search (Current work) board.Aug 4 2016, 10:56 PM
debt closed this task as Resolved.Aug 8 2016, 4:40 PM
debt closed subtask T121541: Create Properly Weighted Language Identification Evaluation Sets for Top N Other Wikis as Resolved.Apr 5 2017, 2:51 PM
Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL