Page MenuHomePhabricator
Create Task
Maniphest T149322

Bucketing & Bonuses for TextCat
Closed, ResolvedPublic
Actions

Description

Look at ways to put languages into buckets for scoring bonuses or penalties.

Three main mathematical approaches are scaling against lowest score, scaling against max possible score, and a per-bucket multiplier.

Main approaches to bucketing are giving boosts to the host language (a priori the most likely), and giving penalties to poorly-performing languages (e.g., infrequent languages that are often confused with the host language).

Also implement for Perl and PHP versions of TextCat.

Details

ProjectBranchLines +/-Subject
wikimedia/textcatmaster+109 -11Add ability to boost liklihood of specified languages
Customize query in gerrit

Related Objects
Search...

Event Timeline

TJones created this task.Oct 27 2016, 3:56 PM
Restricted Application added a project: Discovery-Search. · View Herald TranscriptOct 27 2016, 3:56 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
debt moved this task from needs triage to Up Next on the Discovery-Search board.Oct 27 2016, 8:37 PM
TJones edited projects, added Discovery-Search (Current work); removed Discovery-Search.Dec 19 2016, 7:06 PM
TJones moved this task from Incoming to not in use - please delete on the Discovery-Search (Current work) board.
TJones updated the task description. (Show Details)Jan 9 2017, 9:20 PM

Perl version updated.

gerritbot added a subscriber: gerritbot.Jan 9 2017, 10:17 PM

Change 331399 had a related patch set uploaded (by Tjones):
Add ability to boost liklihood of specified languages

https://gerrit.wikimedia.org/r/331399

gerritbot added a project: Patch-For-Review.Jan 9 2017, 10:17 PM
TJones moved this task from not in use - please delete to Needs review on the Discovery-Search (Current work) board.Jan 9 2017, 11:04 PM
TJones added a comment.Jan 12 2017, 1:19 AM

Write up and analysis is done.

  • Enabling most of the languages that got turned off for too many false positives in the early days can now be turned back on, so we should!
  • Enabling the basic bonus feature, and enabling it for the top 2 languages for each wiki, further improves performance. We should do that!
  • We still need to see how this feature interacts with other potential features being considered—but there's only one left: the unknown n-gram penalty (T151230).
gerritbot added a comment.Jan 17 2017, 10:51 PM

Change 331399 merged by jenkins-bot:
Add ability to boost liklihood of specified languages

https://gerrit.wikimedia.org/r/331399

TJones moved this task from Needs review to Needs Reporting on the Discovery-Search (Current work) board.Feb 7 2017, 5:49 PM
Deskana closed this task as Resolved.Feb 10 2017, 5:27 PM
Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL