Page MenuHomePhabricator

Decide which set of separators we have to use for TextCat ngrams
Closed, ResolvedPublic

Description

The Perl version of TextCat uses only a simple set of non-word characters to separate words - '0-9\s'. However, in natural texts more characters could be separating words and found at the end of a word - such as parentheses, comma, period, etc. We need to test if it makes sense to amend that regular expression or maybe to use some special regexp syntax like \b or \W, to improve model quality.

Related Objects

StatusAssignedTask
ResolvedEBernhardson
Declinedmpopov
ResolvedEBernhardson
Resolvedmpopov
ResolvedEBernhardson
Resolveddebt
OpenNone
ResolvedEBernhardson
ResolvedEBernhardson
ResolvedEBernhardson
Resolveddebt
ResolvedTJones
ResolvedTJones
ResolvedTJones
ResolvedTJones
ResolvedTJones
Resolveddebt
ResolvedAnikethfoss
ResolvedTJones
Resolveddebt
ResolvedSmalyshev
ResolvedTJones
ResolvedTJones
Resolved dpatrick
ResolvedEBernhardson

Event Timeline

Smalyshev assigned this task to TJones.
Smalyshev raised the priority of this task from to Normal.
Smalyshev updated the task description. (Show Details)
Smalyshev added a subscriber: Smalyshev.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 14 2016, 6:49 PM
Smalyshev moved this task from Needs triage to Search on the Discovery board.Jan 15 2016, 12:55 AM

@TJones @Smalyshev Does this block anything specific, or is this just a more along the lines of a general improvement? Would be good to know, so that we can determine whether or not it can be done after we actually get an A/B test out that uses TextCat.

Deskana moved this task from Search to On Sprint Board on the Discovery board.Jan 21 2016, 11:24 PM

We can keep using the minimal set os separators, but the result would (might) be sub-optimal then. So, I think it is useful to do this before we do T123537. Though strictly speaking we could do without it if we had to

We can keep using the minimal set os separators, but the result would (might) be sub-optimal then. So, I think it is useful to do this before we do T123537. Though strictly speaking we could do without it if we had to

Thanks. Marking this as blocking T123537 for now makes sense. We can revisit and maybe remove that requirement if this ends up taking significantly longer than expected.

I've tested the various punctuation options. Long story short, parens hurt, but periods help. More detail:
https://www.mediawiki.org/wiki/User:TJones_(WMF)/Notes/Language_Detection_with_TextCat#Additional_Non-Word_Characters

I'm going to commit the change to the PHP version of TextCat.

Looks great, thanks!

I've tested the various punctuation options. Long story short, parens hurt, but periods help. More detail:
https://www.mediawiki.org/wiki/User:TJones_(WMF)/Notes/Language_Detection_with_TextCat#Additional_Non-Word_Characters
I'm going to commit the change to the PHP version of TextCat.

Great! Can you link to that commit here, so that once it's merged we can close this task?

I got a little over-excited on Friday and got ahead of myself. While the test is done, the language models need to be regenerated with the new separator. I've just done that, but now I'm wrestling with Gerrit. Will link as soon as I can.

Changes committed and awaiting review: https://gerrit.wikimedia.org/r/266437

Merged. (Thanks, for the review, Stas.)

Smalyshev closed this task as Resolved.Jan 26 2016, 11:07 PM