Use the training data created in T118287 for training models for the ES Plugin / Cybozu. Perhaps its difficulties with queries are partly due to the use of general language models. This could also include looking at the internals and seeing if there is any benefit to changing the model size or other internal configuration, including optionally disabling "unhelpful" models (e.g., Romanian when working on enwiki).
Estimate is difficult to make, but it could be time-boxed to 1 or 2 weeks.
Summoning @dcausse — this was originally your idea. Do you have time/interest to work on it? Do you want to call dibs?
Sure,
I think we could do it in 2 pass :
In the end I'd say that cybozu must outperform textcat by a large margin in order to remain a contender:
TextCat PHP implementation offers many advantages :
I'll upload a new version of the plugin with a limited language list so you can give it a try.
As soon as you give me the training set I'll upload a new one.
I realized that we can already limit the language list with a config param:
in /etc/elasticsearch/elasticsearch.yml :
profile: "/langdetect/short-text/" languages: en,es,zh-cn,zh-tw,pt,ar,ru,fa,ko,bn,bg,hi,el,ta,th
I'm not sure about zh-cn and zh-tw, I think that in cirrus we consider both to be Chinese.
@TJones: Do you have a running elastic instance where you could set these settings and run an evaluation?
I can setup the hypothesis testing cluster with it if it's easier.
@dcausse, when I was evaluating the plugin originally, I truncated the language code at the dash, so that zh-cn and zh-tw both count as zh. As I understand it, zhwiki handles both traditional and simplified Chinese.
Looking at my config, I should be able to test locally with a limited language set.
Trey ran 2 other tests:
| profile | F0.5 |
| short-text | 55 |
| short-text with limited language list | 75 |
| input.filtered with limited language list | 81.8 |
While we've seen good improvements TextCat (with the same training data) is better with 83.1.
Building the profiles and running the test was relatively easy, so we can re-test it if needed.
Based on this result I'd suggest to invest more time on TextCat and better training data.