Page MenuHomePhabricator
Create Task
Maniphest T124844

Add textcat to mediawiki vendor libs
Closed, ResolvedPublic
Actions

Description

In order to use textcat for language detection, we need to add Textcat library to Mediawiki vendor repo. The process is described here: https://www.mediawiki.org/wiki/Manual:External_libraries

Details

SubjectRepoBranchLines +/-
Add wikimedia/textcat to vendor repositorymediawiki/vendormaster+751 K -2
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedEBernhardsonT137158 Compile and then resolve issues with TextCat A/B test data
 DeclinedmpopovT134320 Analyse results of TextCat A/B test
 ResolvedEBernhardsonT130321 Disable Schema:Search, since it's outdated and redundant
 ResolvedmpopovT129564 Switch Desktop data collection for dashboards to use TestSearchSatisfaction2 instead of Search schema
 ResolvedEBernhardsonT134319 Turn off TextCat A/B test on the English Wikipedia on or after May 23
 ResolveddebtT134318 Verify data pipeline for TextCat A/B test on English Wikipedia
 OpenNoneT118278 [EPIC] Improve Language Identification for use in Cirrus Search
 ResolvedEBernhardsonT121542 Write and deploy an A/B Test on enwiki using TextCat for Language Identification
 ResolvedEBernhardsonT124844 Add textcat to mediawiki vendor libs
 ResolvedEBernhardsonT121543 Do an A/B Tests on Other Wikis with TextCat for Language Identification
 ResolveddebtT121541 Create Properly Weighted Language Identification Evaluation Sets for Top N Other Wikis
 ResolvedTJonesT121539 Create Balanced Language Identification Evaluation Set for Top N Wikis by Query Volume
 ResolvedTJonesT132466 Lang ID Eval Sets for Italian, German, Spanish, and French
 ResolvedTJonesT134431 Re-Optimize Italian, German, Spanish, and French TextCat Languages by Recall
 ResolvedTJonesT138315 Lang ID Eval Sets for English, Russian, Japanese, Portuguese
 ResolvedTJonesT142413 Deploy recommended languages for Russian, Japanese, and Portuguese
 ResolveddebtT143355 request translations for 'showing results from'
 ResolvedAnikethfossT145926 [[MediaWiki:Search-interwiki-results-acewiki/fi]] typo: "Acehnese" instead of "Achinese"
 ResolvedTJonesT142140 Lang ID Eval Set for Dutch
 ResolveddebtT143354 ask for translations for 'showing results from' (Polish, Dutch, Arabic and Chinese)
 ResolvedSmalyshevT121538 Convert TextCat to PHP Library for Language Identification in Cirrus Search
 ResolvedTJonesT123537 Generate wikitext-based and query-based language models for TextCat
 ResolvedTJonesT123651 Decide which set of separators we have to use for TextCat ngrams
 Resolved• dpatrickT123558 Security review for TextCat library
 ResolvedEBernhardsonT137163 Part Deux: TextCat A/B test for Language Identification - specification

Event Timeline

Smalyshev created this task.Jan 26 2016, 11:12 PM
Smalyshev raised the priority of this task from to Medium.
Smalyshev updated the task description. (Show Details)
Smalyshev added a project: Discovery-ARCHIVED.
Smalyshev subscribed.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 26 2016, 11:12 PM
Smalyshev added subtasks: T123558: Security review for TextCat library, T121538: Convert TextCat to PHP Library for Language Identification in Cirrus Search.Jan 26 2016, 11:12 PM
Smalyshev added a parent task: T121542: Write and deploy an A/B Test on enwiki using TextCat for Language Identification.
Smalyshev moved this task from Needs triage to Search on the Discovery-ARCHIVED board.Jan 27 2016, 7:12 AM
• Deskana added a project: Discovery-Search (Current work).Jan 28 2016, 5:47 PM
• Deskana set Security to None.
• Deskana subscribed.

Given that both the task that this blocks and the task that this is blocked by are in Discovery-Search (Current work), I think this one probably should be as well. :-)

• Deskana moved this task from Search to On Sprint Board on the Discovery-ARCHIVED board.Jan 28 2016, 5:48 PM
Smalyshev closed subtask T121538: Convert TextCat to PHP Library for Language Identification in Cirrus Search as Resolved.Jan 28 2016, 6:03 PM
• Deskana mentioned this in T121538: Convert TextCat to PHP Library for Language Identification in Cirrus Search.Jan 28 2016, 6:09 PM
Smalyshev added a subtask: T123537: Generate wikitext-based and query-based language models for TextCat.Jan 28 2016, 7:10 PM
EBernhardson claimed this task.Feb 3 2016, 12:41 AM
EBernhardson moved this task from Incoming to not in use - please delete on the Discovery-Search (Current work) board.
gerritbot subscribed.Feb 3 2016, 12:49 AM

Change 268028 had a related patch set uploaded (by EBernhardson):
Add wikimedia/textcat to vendor repository

https://gerrit.wikimedia.org/r/268028

gerritbot added a project: Patch-For-Review.Feb 3 2016, 12:49 AM
EBernhardson moved this task from not in use - please delete to Needs review on the Discovery-Search (Current work) board.Feb 3 2016, 12:52 AM
EBernhardson added a comment.Feb 3 2016, 1:06 AM

@Smalyshev one thing i noticed while setting this up is that we are pulling from the github repo. Should we adjust the packagist configuration such that it pulls from gerrit? This change would be inline with the security review.

Smalyshev added a comment.Feb 3 2016, 1:07 AM

I think all wikimedia libs go through github there?

Smalyshev closed subtask T123537: Generate wikitext-based and query-based language models for TextCat as Resolved.Feb 3 2016, 1:08 AM
bd808 subscribed.Feb 3 2016, 1:30 AM
In T124844#1992606, @Smalyshev wrote:

I think all wikimedia libs go through github there?

We have not attempted to use a private package repo for anything in mediawiki/vendor.git up to this point. It should be technically possible, but the current process of manual gerrit review for changes to the repo has been sufficient.

Smalyshev added a comment.Feb 3 2016, 1:30 AM

Thanks @bd808, that's what I thought.

gerritbot added a comment.Feb 9 2016, 10:34 PM

Change 268028 merged by jenkins-bot:
Add wikimedia/textcat to vendor repository

https://gerrit.wikimedia.org/r/268028

Smalyshev closed this task as Resolved.Feb 9 2016, 11:12 PM
Smalyshev closed subtask T123558: Security review for TextCat library as Resolved.Feb 11 2016, 7:34 PM
• Deskana moved this task from Needs review to Needs Reporting on the Discovery-Search (Current work) board.Feb 11 2016, 11:07 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits