Page MenuHomePhabricator

Wikipedia-Text–Based Language Models for Language Identification
Closed, ResolvedPublic

Description

Though T118287 shows that query-based text models work better than wiki-text–based language models, getting good query data can be hard. So, we should see if Wikipedia-based language models for languages with crappy training data do better. (could obviate the need for T121544 or T121546 in some cases; could be tested via re-test of data in T121539 or T121541 with current best language identification module)

Need list of target languages to work on (based on T121539).

Estimate: 2-4 hours per language to create corpuse, do minimal cleanup, and build & test model(s).

Related Objects

Event Timeline

TJones created this task.Dec 15 2015, 5:51 PM
TJones raised the priority of this task from to Needs Triage.
TJones updated the task description. (Show Details)
TJones added a project: CirrusSearch.
TJones added a subscriber: TJones.
Restricted Application added a project: Discovery. · View Herald TranscriptDec 15 2015, 5:51 PM
Restricted Application added subscribers: StudiesWorld, Aklapper. · View Herald Transcript
Deskana triaged this task as Normal priority.Jan 27 2016, 11:09 PM
Deskana moved this task from Needs triage to Search on the Discovery board.
Deskana added a subscriber: Deskana.
TJones added a comment.Feb 1 2016, 9:30 PM

I've generated these models based on lightly cleaned up wiki-text and submitted them for review.

The models haven't been vetted, but T121539 isn't going to cover all 70 of these languages anyway.

Restricted Application added a project: Discovery-Search. · View Herald TranscriptMay 11 2016, 10:40 PM
debt moved this task from Up Next to later on... on the Discovery-Search board.Jun 2 2016, 10:06 PM
debt added a subscriber: debt.Aug 4 2016, 7:00 PM

From a conversation with @TJones:

This is a good place to spend time. Wikitext models are generally easy to create by pulling articles from the relevant Wikipedia for training data. The bang-for-your-buck is probably high enough to do this for all of the medium-effort section of the long-tail (many are already done)

TJones claimed this task.EditedAug 4 2016, 8:43 PM

As we are focusing on the top 25 languages by volume from the dashboard, we have either query-based models or wiki-text–based models for all 25. (Only Finnish, Hungarian, Norwegian, and Tagalog don't have query-based models.)

BTW, the top 25 are: English, German, Spanish, Portuguese, Russian, French, Italian, Japanese, Polish, Arabic, Chinese, Dutch, Turkish, Swedish, Persian, Czech, Vietnamese, Indonesian, Korean, Finnish, Hebrew, Tagalog, Thai, Norwegian, and Hungarian.)

If we need to create new wiki-text–based models while working on the current batch of language wikipedias—say, because there's a new language that's used a lot on one of these wikis—I can spin up a new subtask to build that wiki-text model.

So, I say mark this one done.

Moving to done column on sprint board, based on @TJones's comment above!

debt closed this task as Resolved.Aug 8 2016, 4:40 PM