Page MenuHomePhabricator
Create Task
Maniphest T336683

Enable MinT support for languages with no Wikipedia yet
Closed, ResolvedPublic
Actions

Description

MinT supports languages for which there is no Wikipedia yet. Although MinT won't be immediately usable in Wikipedia, those communities can still use the service to translate other content to their languages.

These are the selected languages to enable based on NLLB-200 supported languages:

  1. Mesopotamian Arabic (acm) ✅
  2. Ta’izzi-Adeni Arabic (acq) ✅
  3. Tunisian Arabic (aeb) ✅
  4. South Levantine Arabic (ajp) ✅
  5. North Levantine Arabic (apc) ✅
  6. Najdi Arabic (ars) ✅
  7. Bemba (bem) ✅
  8. Chokwe (cjk) ✅
  9. Dyula (dyu) ✅
  10. Chhattisgarhi (hne) ✅
  11. Jingpho (kac) ✅
  12. Kamba (kam) ✅
  13. Central Kanuri (knc) Enable support for Central Kanuri (knc) and the more general Kanuri (kr) language ✅
  14. Kabuverdianu (kea) ✅
  15. Kimbundu (kmb) ✅
  16. Luba-Kasai (lua) ✅
  17. Luo (luo) ✅
  18. Mizo (lus) ✅
  19. Magahi (mag) ✅
  20. Mossi (mos) ✅
  21. Nuer (nus) ✅
  22. Tamasheq (taq) ✅
  23. Central Atlas Tamazight (tzm) ✅
  24. Umbundu (umb) ✅
  25. Fon (fon) ✅ Fon has graduated from incubator and there is a Wikipedia now

The IndicTrans2 model also supports the following languages without a Wikipedia (enabled as part of T337656):

  • Bodo (brx/brx_Deva) – ✅
  • Dogri (doi/doi_Deva) – ✅

__
See also T89089: Make ContentTranslation work in the Wikimedia Incubator

Details

SubjectRepoBranchLines +/-
Update MinT to 2023-10-16-101614-productionoperations/deployment-chartsmaster+1 -1
config: Enable Kanuri as kr language code toomediawiki/services/machinetranslationmaster+1 -0
Update MinT to 2023-09-04-051105-productionoperations/deployment-chartsmaster+1 -1
config: Fix language code for Mizomediawiki/services/machinetranslationmaster+2 -1
config: Fix language code for Mizomediawiki/services/cxservermaster+1 -1
config: Add missing language codes and enable taq languagemediawiki/services/machinetranslationmaster+7 -3
Update MinT to 2023-08-14-091403-productionoperations/deployment-chartsmaster+1 -1
Update cxserver to 2023-08-14-091804-productionoperations/deployment-chartsmaster+1 -1
Enable MinT support for languages with no Wikipedia yetmediawiki/services/cxservermaster+26 -0
Enable Kanuri (kr/knc) languagemediawiki/services/machinetranslationmaster+3 -2
Customize query in gerrit

Related Objects

Event Timeline

Pginer-WMF created this task.May 15 2023, 3:02 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 15 2023, 3:02 PM
Pginer-WMF triaged this task as Medium priority.May 15 2023, 3:03 PM
Pginer-WMF mentioned this in T326578: Enable NLLB-200 for languages lacking other Machine Translation options.May 15 2023, 3:16 PM
Bugreporter updated the task description. (Show Details)May 15 2023, 3:44 PM
Bugreporter subscribed.May 15 2023, 3:48 PM

Note Central Kanuri is one of major dialect of Kanuri (kr), which has a Wikipedia, but was closed.

Pginer-WMF updated the task description. (Show Details)May 16 2023, 7:48 AM
Pginer-WMF added a comment.May 16 2023, 8:00 AM
In T336683#8851205, @Bugreporter wrote:

Note Central Kanuri is one of major dialect of Kanuri (kr), which has a Wikipedia, but was closed.

Thanks for the info @Bugreporter. For cases such as this, where the NLLB-200 model indicates the specific variant of the language, it is a bit unclear which code to use if there is no Wikipedia for either the more general language code or the specific variant. In this particular case, I guess it makes sense to support Kanuri with the kr code.

We may need to review the list to make sure whether other languages in the above list may be in a similar situation, and decide the best code (or codes) to use.

Pginer-WMF updated the task description. (Show Details)May 16 2023, 8:01 AM
Pginer-WMF updated the task description. (Show Details)May 16 2023, 8:08 AM
Pginer-WMF moved this task from Backlog to Adding languages on the MinT board.May 16 2023, 8:53 AM
Pginer-WMF updated the task description. (Show Details)Jun 13 2023, 8:06 AM
Pginer-WMF mentioned this in T340953: Enable MinT for all the remaining languages supported by NLLB-200.Jul 3 2023, 10:38 AM
Pginer-WMF updated the task description. (Show Details)Jul 3 2023, 1:18 PM
Pginer-WMF updated the task description. (Show Details)
Pginer-WMF added a comment.Aug 2 2023, 9:50 AM

When comparing the list in the description with the language-data library, all the language codes are there except for Najdi Arabic (ars).

When comparing the list in the description with this localization directory, these seem to be the languages with most activity in translatewiki.net:

  • Mesopotamian Arabic (acm)
  • Mossi (mos)
  • Mizo (lus)
  • Magahi (mag)
  • Kanuri (kr), which can be supported with Central Kanuri (knc)
  • Central Atlas Tamazight (tzm)
  • Fon (fon)
  • Kabuverdianu (kea)
Pginer-WMF assigned this task to KartikMistry.Aug 2 2023, 9:53 AM
Pginer-WMF updated the task description. (Show Details)
Pginer-WMF added a project: Language-Team (Language-2023-July-September).
Pginer-WMF moved this task from Quarter Backlog to Priority: Translation on the Language-Team (Language-2023-July-September) board.
Pginer-WMF updated the task description. (Show Details)
gerritbot added a comment.Aug 14 2023, 6:05 AM

Change 948241 had a related patch set uploaded (by KartikMistry; author: KartikMistry):

[mediawiki/services/machinetranslation@master] Enable Kanuri (kr/knc) language

https://gerrit.wikimedia.org/r/948241

gerritbot added a project: Patch-For-Review.Aug 14 2023, 6:05 AM
KartikMistry moved this task from Priority: Translation to In Review on the Language-Team (Language-2023-July-September) board.Aug 14 2023, 6:07 AM
gerritbot added a comment.Aug 14 2023, 6:09 AM

Change 948242 had a related patch set uploaded (by KartikMistry; author: KartikMistry):

[mediawiki/services/cxserver@master] Enable MinT support for languages with no Wikipedia yet

https://gerrit.wikimedia.org/r/948242

gerritbot added a comment.Aug 14 2023, 9:13 AM

Change 948241 merged by jenkins-bot:

[mediawiki/services/machinetranslation@master] Enable Kanuri (kr/knc) language

https://gerrit.wikimedia.org/r/948241

gerritbot added a comment.Aug 14 2023, 9:17 AM

Change 948242 merged by jenkins-bot:

[mediawiki/services/cxserver@master] Enable MinT support for languages with no Wikipedia yet

https://gerrit.wikimedia.org/r/948242

Maintenance_bot removed a project: Patch-For-Review.Aug 14 2023, 9:31 AM
gerritbot added a comment.Aug 17 2023, 4:27 AM

Change 949619 had a related patch set uploaded (by KartikMistry; author: KartikMistry):

[operations/deployment-charts@master] Update cxserver to 2023-08-14-091804-production

https://gerrit.wikimedia.org/r/949619

gerritbot added a project: Patch-For-Review.Aug 17 2023, 4:27 AM
gerritbot added a comment.Aug 17 2023, 8:06 AM

Change 949619 merged by jenkins-bot:

[operations/deployment-charts@master] Update cxserver to 2023-08-14-091804-production

https://gerrit.wikimedia.org/r/949619

Maintenance_bot removed a project: Patch-For-Review.Aug 17 2023, 8:10 AM
Stashbot added a comment.Aug 17 2023, 8:31 AM

Mentioned in SAL (#wikimedia-operations) [2023-08-17T08:31:49Z] <kart_> Updated cxserver to 2023-08-14-091804-production (T336683, T343211)

Stashbot mentioned this in T343211: Enable Content and Section translation on 7 Wikipedias.Aug 17 2023, 8:31 AM
KartikMistry updated the task description. (Show Details)Aug 18 2023, 2:49 AM
gerritbot added a comment.Aug 18 2023, 2:54 AM

Change 950063 had a related patch set uploaded (by KartikMistry; author: KartikMistry):

[operations/deployment-charts@master] Update MinT to 2023-08-14-091403-production

https://gerrit.wikimedia.org/r/950063

gerritbot added a project: Patch-For-Review.Aug 18 2023, 2:54 AM
gerritbot added a comment.Aug 21 2023, 6:04 AM

Change 950063 merged by jenkins-bot:

[operations/deployment-charts@master] Update MinT to 2023-08-14-091403-production

https://gerrit.wikimedia.org/r/950063

Maintenance_bot removed a project: Patch-For-Review.Aug 21 2023, 6:10 AM
Stashbot added a comment.Aug 21 2023, 6:28 AM

Mentioned in SAL (#wikimedia-operations) [2023-08-21T06:28:57Z] <kart_> Update MinT to 2023-08-14-091403-production (T336683)

KartikMistry updated the task description. (Show Details)Aug 21 2023, 6:29 AM
KartikMistry moved this task from In Review to Needs QA on the Language-Team (Language-2023-July-September) board.
Pginer-WMF added a comment.Aug 28 2023, 9:45 AM

Testing these languages in this test instance I found that most worked as expected. However, some languages were not listed, and other languages were listed but they got stuck in the translation process without providing a response.

Not listed:

  • North Levantine Arabic (apc)
  • Najdi Arabic (ars)

Listed, but not responding:

  • Kanuri (kr)
  • Mizo (lus)
  • Tamasheq (taq)
KartikMistry changed the task status from Open to In Progress.Aug 28 2023, 11:43 AM
KartikMistry moved this task from Needs QA to In Progress on the Language-Team (Language-2023-July-September) board.
gerritbot added a comment.Aug 28 2023, 11:53 AM

Change 952844 had a related patch set uploaded (by KartikMistry; author: KartikMistry):

[mediawiki/services/cxserver@master] config: Fix language code for Mizo

https://gerrit.wikimedia.org/r/952844

gerritbot added a project: Patch-For-Review.Aug 28 2023, 11:53 AM

Change 952845 had a related patch set uploaded (by KartikMistry; author: KartikMistry):

[mediawiki/services/machinetranslation@master] config: Add missing language codes and enable taq language

https://gerrit.wikimedia.org/r/952845

KartikMistry added a comment.Aug 28 2023, 11:55 AM
In T336683#9122784, @Pginer-WMF wrote:

Testing these languages in this test instance I found that most worked as expected. However, some languages were not listed, and other languages were listed but they got stuck in the translation process without providing a response.

Not listed:

  • North Levantine Arabic (apc)
  • Najdi Arabic (ars)

Listed, but not responding:

  • Kanuri (kr)
  • Mizo (lus)
  • Tamasheq (taq)

Thanks, Pau. I've submitted patches to fix the above issues, but I'm still checking about Kanuri (kr).

Pginer-WMF mentioned this in T339926: The NLLB-200 MT engine in MinT returns standard Arabic translation instead of Moroccan Darija in Moroccan Arabic Wikipedia.Aug 29 2023, 9:11 AM
gerritbot added a comment.Aug 30 2023, 8:18 AM

Change 952845 merged by jenkins-bot:

[mediawiki/services/machinetranslation@master] config: Add missing language codes and enable taq language

https://gerrit.wikimedia.org/r/952845

gerritbot added a comment.Aug 31 2023, 5:23 AM

Change 952844 abandoned by KartikMistry:

[mediawiki/services/cxserver@master] config: Fix language code for Mizo

Reason:

Need to fix in the MinT.

https://gerrit.wikimedia.org/r/952844

Maintenance_bot removed a project: Patch-For-Review.Aug 31 2023, 5:30 AM
gerritbot added a comment.Aug 31 2023, 5:41 AM

Change 953754 had a related patch set uploaded (by KartikMistry; author: KartikMistry):

[mediawiki/services/machinetranslation@master] config: Fix language code for Mizo

https://gerrit.wikimedia.org/r/953754

gerritbot added a project: Patch-For-Review.Aug 31 2023, 5:41 AM
gerritbot added a comment.Aug 31 2023, 6:11 AM

Change 953754 merged by jenkins-bot:

[mediawiki/services/machinetranslation@master] config: Fix language code for Mizo

https://gerrit.wikimedia.org/r/953754

Maintenance_bot removed a project: Patch-For-Review.Aug 31 2023, 6:30 AM
gerritbot added a comment.Aug 31 2023, 12:16 PM

Change 954005 had a related patch set uploaded (by KartikMistry; author: KartikMistry):

[operations/deployment-charts@master] Update MinT to 2023-08-31-061147-production

https://gerrit.wikimedia.org/r/954005

gerritbot added a project: Patch-For-Review.Aug 31 2023, 12:16 PM
gerritbot added a comment.Sep 5 2023, 5:12 AM

Change 954005 merged by jenkins-bot:

[operations/deployment-charts@master] Update MinT to 2023-09-04-051105-production

https://gerrit.wikimedia.org/r/954005

Maintenance_bot removed a project: Patch-For-Review.Sep 5 2023, 5:30 AM
Stashbot added a comment.Sep 5 2023, 5:55 AM

Mentioned in SAL (#wikimedia-operations) [2023-09-05T05:55:09Z] <kart_> Updated MinT to 2023-09-04-051105-production (T336683)

Pginer-WMF added a comment.Sep 7 2023, 9:23 AM
In T336683#9123160, @KartikMistry wrote:
In T336683#9122784, @Pginer-WMF wrote:

Testing these languages in this test instance I found that most worked as expected. However, some languages were not listed, and other languages were listed but they got stuck in the translation process without providing a response.

Not listed:

  • North Levantine Arabic (apc)
  • Najdi Arabic (ars)

Listed, but not responding:

  • Kanuri (kr)
  • Mizo (lus)
  • Tamasheq (taq)

Thanks, Pau. I've submitted patches to fix the above issues, but I'm still checking about Kanuri (kr).

Thanks! I just checked, and most of the issues are resolved, the only one pending is for Kanuri (kr) being listed but not responding.

Pginer-WMF mentioned this in T331505: Self hosted machine translation service.Sep 7 2023, 9:56 AM
Pginer-WMF edited projects, added Language-Team (Language-2023-October-December); removed Language-Team (Language-2023-July-September).Oct 2 2023, 9:01 AM
Pginer-WMF moved this task from Quarter Backlog to In Progress on the Language-Team (Language-2023-October-December) board.
Pginer-WMF updated the task description. (Show Details)Oct 5 2023, 8:57 AM
gerritbot added a comment.Oct 12 2023, 12:24 PM

Change 965508 had a related patch set uploaded (by KartikMistry; author: KartikMistry):

[mediawiki/services/machinetranslation@master] config: Enable Kanuri as kr language code too

https://gerrit.wikimedia.org/r/965508

gerritbot added a project: Patch-For-Review.Oct 12 2023, 12:24 PM
KartikMistry moved this task from In Progress to In Review on the Language-Team (Language-2023-October-December) board.Oct 12 2023, 12:25 PM
gerritbot added a comment.Oct 16 2023, 10:16 AM

Change 965508 merged by jenkins-bot:

[mediawiki/services/machinetranslation@master] config: Enable Kanuri as kr language code too

https://gerrit.wikimedia.org/r/965508

Maintenance_bot removed a project: Patch-For-Review.Oct 16 2023, 10:31 AM
gerritbot added a comment.Oct 16 2023, 11:46 AM

Change 966170 had a related patch set uploaded (by KartikMistry; author: KartikMistry):

[operations/deployment-charts@master] Update MinT to 2023-10-16-101614-production

https://gerrit.wikimedia.org/r/966170

gerritbot added a project: Patch-For-Review.Oct 16 2023, 11:46 AM
gerritbot added a comment.Oct 17 2023, 5:17 AM

Change 966170 merged by jenkins-bot:

[operations/deployment-charts@master] Update MinT to 2023-10-16-101614-production

https://gerrit.wikimedia.org/r/966170

Maintenance_bot removed a project: Patch-For-Review.Oct 17 2023, 5:30 AM
Stashbot mentioned this in T333969: Enable Opus models for languages lacking other Machine Translation options.Oct 17 2023, 5:59 AM
Stashbot mentioned this in T348097: Twi is listed as Akan in the MinT translation interface.

Mentioned in SAL (#wikimedia-operations) [2023-10-17T05:59:44Z] <kart_> Update MinT to 2023-10-16-101614-production (T333969, T336683, T348097)

KartikMistry added a comment.Oct 17 2023, 6:01 AM

Thanks! I just checked, and most of the issues are resolved, the only one pending is for Kanuri (kr) being listed but not responding.

@Pginer-WMF This should be fixed now.

KartikMistry moved this task from In Review to Done on the Language-Team (Language-2023-October-December) board.Oct 17 2023, 6:01 AM
Pginer-WMF added a comment.Oct 17 2023, 7:49 AM
In T336683#9256352, @KartikMistry wrote:

Thanks! I just checked, and most of the issues are resolved, the only one pending is for Kanuri (kr) being listed but not responding.

@Pginer-WMF This should be fixed now.

Perfect. It works now!

translate.wmcloud.org_text(Wiki Tablet).png (768×1 px, 83 KB)

Pginer-WMF closed this task as Resolved.Oct 17 2023, 7:50 AM
Ogalihillary subscribed.Nov 20 2023, 7:19 PM

I wish Igala language is part of the languages added here, though still incubator but machine in translation will be so helpful.

Pginer-WMF added a comment.Nov 21 2023, 10:26 AM
In T336683#9346565, @Ogalihillary wrote:

I wish Igala language is part of the languages added here, though still incubator but machine in translation will be so helpful.

In order to enable support Igala we need a machine translation model that supports the language. Unfortunately the models currently integrated in MinT do not seem to support Igala. There are some ways to help change that:

  • Find a freely licensed translation model that supports the language. In this way, we could integrate it in MinT.
  • Help expand the multilingual content available in Igala for future models to be creaed. This can be done by:
    • Using Content Translation to translate Wikipedia articles in Igala, once Igala Wikipedia graduates from Incubator (the tool is not available at the incubator stage).
    • Contacting the Tatoeba project to enable support for Igala, and provide translations of sentences.
    • Share any freely licensed multilingual data that includes Igala content with the opus project.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits