I'm not sure what is this about. Is it supposed to be in CX?

Implementation wise it should be separate API module or service and not tied to CX.

A language detection service can be relevant in the context of the upcoming hosting of machine translation (T331505). Integration with the services would open the possibility to have the source language as an optional parameter. In such case, the source language would be detected automatically using the service proposed in the current ticket.

In T334465: MinT: Detect language of source content automatically we used pycld2 which is a simple old library for language detection since the usecase was a silent attempt to detect languages while users make their choice on top of it. This library can detect only ~83 languages.

For a general purpose language detection service for large set of languages, the current known approach is fasttext language classification model that can detect 176 languages.

Change 908223 had a related patch set uploaded (by Santhosh; author: Santhosh):

[mediawiki/services/machinetranslation@master] Use fasttext based language detection

https://gerrit.wikimedia.org/r/908223

To support more languages, we can follow the documentation at https://fasttext.cc/blog/2017/10/02/blog-post.html
However, this require a large training dataset that has samples for all the languages we want to support. For example, tum is not supported fasttext. So we need a dataset of tum language sentences(about 100K for example).

So this becomes a task to prepare dataset that has large number of sentence samples for every language support. Such a dataset would be useful in many contexts too. It is not difficult to create it - may be a crawler with a sentence splitter is a good starting point.

From my experience fasttext is quite fast and can do this training in sufficiently capable CPUs.

Change 932828 had a related patch set uploaded (by Santhosh; author: Santhosh):

[machinelearning/liftwing/inference-services@main] Add language identification service

https://gerrit.wikimedia.org/r/932828

I came across this paper recently https://arxiv.org/pdf/2305.13820.pdf which explains the process of creating larger language identification models. They also provide a model that covers 201 languages

• Source: https://github.com/laurieburchell/open-lid-dataset
• Model: https://data.statmt.org/lid/lid201-model.bin.gz
• Model license: the GNU General Public License v3.0.

As we have LiftWing that helps to easily host these models, I submitted a patch to add the service.

@santhosh Hi! Very happy to see this request :) Do you mind to open a separate task as indicated in https://wikitech.wikimedia.org/wiki/Machine_Learning/LiftWing#Hosting_a_model ?

Our way of doing things is to assign a ML Engineer to every request, so they can discuss the use case with the requester and follow up accordingly (for example, to set up CI, upload the model binary, etc..)

Change 933092 had a related patch set uploaded (by Ilias Sarantopoulos; author: Ilias Sarantopoulos):

[integration/config@master] inference-services: add langid CI pipelines

https://gerrit.wikimedia.org/r/933092

Change 933092 merged by jenkins-bot:

[integration/config@master] inference-services: add langid CI pipelines

https://gerrit.wikimedia.org/r/933092

Change 908223 abandoned by Santhosh:

[mediawiki/services/machinetranslation@master] language detection: Use fasttext

Reason:

This will be a new service hosted in lift wing

https://gerrit.wikimedia.org/r/908223

We have the service in production: https://api.wikimedia.org/wiki/Lift_Wing_API/Reference/Get_language_identification_prediction