Page MenuHomePhabricator
Create Task
Maniphest T339896

Enable MinT for all languages supported by IndicTrans2
Closed, ResolvedPublic
Actions

Description

After the initial exploration of IndicTrans2 (T337656), we want to enable MinT (list of current language support) for all the languages supported:

  1. Assamese (as/asm_Beng) — ✅ Enabled as part of T337656
  2. Bangla (bn/ben_Beng) — ✅ Enabled as part of T337656
  3. Hindi (hi/hin_Deva) — ✅ Enabled as part of T337656
  4. Kashmiri (ks/kas_Arab) — ✅ Enabled as part of T337656
  5. Santali (sat/sat_Olck) — ✅ Enabled as part of T337656
  6. Goan (gom/gom_Deva)
  7. Gujarati (gu/guj_Gujr)
  8. Kannada (kn/kan_Knda)
  9. Maithili (mai/mai_Deva)
  10. Malayalam (ml/mal_Mlym)
  11. Manipuri (mni/mni_Beng & mni_Mtei)
  12. Marathi (mr/mar_Deva)
  13. Nepali (ne/npi_Deva)
  14. Oriya (or/ory_Orya)
  15. Panjabi (pa/pan_Guru)
  16. Sanskrit (sa/san_Deva)
  17. Sindhi (sd/snd_Arab & snd_Deva)
  18. Tamil (ta/tam_Taml)
  19. Telugu (te/tel_Telu)
  20. Urdu (ur/urd_Arab)
  21. Bodo (brx/brx_Deva) No wiki yet. Enable to have it ready for the future (unless that could be problematic)
  22. Dogri (doi/doi_Deva) No wiki yet. Enable to have it ready for the future (unless that could be problematic)

Steps:

  • Enable MinT for selected languages
  • Communicate with affected communities (asking also about which service they prefer as the default).

Details

Other Assignee
UOzurumba
Related Changes in Gerrit:
SubjectRepoBranchLines +/-
Update MinT to 2023-06-27-053706-productionoperations/deployment-chartsmaster+1 -1
Add gom code to WIKI2NLLBCODESmediawiki/services/machinetranslationmaster+1 -0
Update cxserver to 2023-06-26-050753-productionoperations/deployment-chartsmaster+1 -1
MinT: Update IndicTrans2 only supported pairsmediawiki/services/cxservermaster+9 -3
Update cxserver to 2023-06-21-112200-productionoperations/deployment-chartsmaster+1 -1
Enable MinT for all languages supported by IndicTrans2mediawiki/services/cxservermaster+17 -0
Customize query in gerrit

Related Objects

Event Timeline

Pginer-WMF created this task.Jun 20 2023, 9:15 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 20 2023, 9:15 AM
Pginer-WMF assigned this task to KartikMistry.Jun 20 2023, 9:15 AM
Pginer-WMF triaged this task as Medium priority.
Pginer-WMF updated the task description. (Show Details)Jun 20 2023, 9:18 AM
Pginer-WMF moved this task from Quarter Backlog to Priority: Unified Content & Section Translation on the Language-Team (Language-2023-April-June) board.Jun 20 2023, 9:33 AM
Pginer-WMF moved this task from Backlog to Adding languages on the MinT board.
gerritbot added a comment.Jun 21 2023, 7:58 AM

Change 931870 had a related patch set uploaded (by KartikMistry; author: KartikMistry):

[mediawiki/services/cxserver@master] Enable MinT for all languages supported by IndicTrans2

https://gerrit.wikimedia.org/r/931870

gerritbot added a project: Patch-For-Review.Jun 21 2023, 7:58 AM
KartikMistry moved this task from Priority: Unified Content & Section Translation to In Review on the Language-Team (Language-2023-April-June) board.Jun 21 2023, 7:58 AM
KartikMistry mentioned this in T339105: Enable MinT for languages where mobile translation is supported.Jun 21 2023, 8:11 AM
gerritbot added a comment.Jun 21 2023, 11:21 AM

Change 931870 merged by jenkins-bot:

[mediawiki/services/cxserver@master] Enable MinT for all languages supported by IndicTrans2

https://gerrit.wikimedia.org/r/931870

Maintenance_bot removed a project: Patch-For-Review.Jun 21 2023, 11:31 AM
gerritbot added a comment.Jun 22 2023, 4:04 AM

Change 932042 had a related patch set uploaded (by KartikMistry; author: KartikMistry):

[operations/deployment-charts@master] Update cxserver to 2023-06-21-112200-production

https://gerrit.wikimedia.org/r/932042

gerritbot added a project: Patch-For-Review.Jun 22 2023, 4:04 AM
gerritbot added a comment.Jun 22 2023, 6:30 AM

Change 932042 merged by jenkins-bot:

[operations/deployment-charts@master] Update cxserver to 2023-06-21-112200-production

https://gerrit.wikimedia.org/r/932042

Maintenance_bot removed a project: Patch-For-Review.Jun 22 2023, 6:30 AM
Stashbot added a comment.Jun 22 2023, 6:39 AM

Mentioned in SAL (#wikimedia-operations) [2023-06-22T06:39:01Z] <kart_> Updated cxserver to 2023-06-21-112200-production (T339896, T338123)

Stashbot mentioned this in T338123: Enable MinT, Content and Section Translation for a 4th group of languages previously lacking machine translation.Jun 22 2023, 6:39 AM
KartikMistry updated the task description. (Show Details)Jun 22 2023, 6:39 AM
KartikMistry updated Other Assignee, added: UOzurumba.
KartikMistry added a subscriber: UOzurumba.Jun 22 2023, 6:41 AM

@UOzurumba We can notify these communities about the availability of the MinT service and ask for their default preference. We will adjust the preference if we will receive any feedback on the other task. Once notification is done, we can mark this task as complete.

KartikMistry moved this task from In Review to In Progress on the Language-Team (Language-2023-April-June) board.Jun 22 2023, 6:42 AM
Pginer-WMF added a comment.Jun 22 2023, 8:11 AM

@KartikMistry, an adjustment in the configuration may be needed for the following languages:

  • Dogri (doi/doi_Deva)
  • Bodo (brx/brx_Deva)
  • Goan (gom/gom_Deva)

For these languages the configuration should capture only the language pairs involving English (not all combinations between them). The reason is that IndicTrans2 only supports translations from/to English and NLLB-200 (which supports all combinations) could not be used as a fallback for the above three languages since they are not supported by the model.

gerritbot added a comment.Jun 22 2023, 8:15 AM

Change 932172 had a related patch set uploaded (by KartikMistry; author: KartikMistry):

[mediawiki/services/cxserver@master] MinT: Update IndicTrans2 only supported pairs

https://gerrit.wikimedia.org/r/932172

gerritbot added a project: Patch-For-Review.Jun 22 2023, 8:15 AM
gerritbot added a comment.Jun 22 2023, 10:03 AM

Change 932172 merged by jenkins-bot:

[mediawiki/services/cxserver@master] MinT: Update IndicTrans2 only supported pairs

https://gerrit.wikimedia.org/r/932172

Maintenance_bot removed a project: Patch-For-Review.Jun 22 2023, 10:11 AM
UOzurumba added a comment.Jun 22 2023, 10:56 PM
In T339896#8954835, @KartikMistry wrote:

@UOzurumba We can notify these communities about the availability of the MinT service and ask for their default preference. We will adjust the preference if we will receive any feedback on the other task. Once notification is done, we can mark this task as complete.

Thank you, @KartikMistry, I will make the announcement on Monday.

UOzurumba added a project: MoveComms-Support (Apr-Jun-2023).Jun 22 2023, 11:13 PM
gerritbot added a comment.Jun 26 2023, 5:13 AM

Change 932683 had a related patch set uploaded (by KartikMistry; author: KartikMistry):

[operations/deployment-charts@master] Update cxserver to 2023-06-26-050753-production

https://gerrit.wikimedia.org/r/932683

gerritbot added a project: Patch-For-Review.Jun 26 2023, 5:13 AM
gerritbot added a comment.Jun 26 2023, 6:09 AM

Change 932683 merged by jenkins-bot:

[operations/deployment-charts@master] Update cxserver to 2023-06-26-050753-production

https://gerrit.wikimedia.org/r/932683

Maintenance_bot removed a project: Patch-For-Review.Jun 26 2023, 6:10 AM
Stashbot added a comment.Jun 26 2023, 6:28 AM

Mentioned in SAL (#wikimedia-operations) [2023-06-26T06:28:05Z] <kart_> Updated cxserver to 2023-06-26-050753-production (T340236, T339896)

Stashbot mentioned this in T340236: MinT translates to English when Hindi-Santali or any other language-Santali is selected.Jun 26 2023, 6:28 AM
KartikMistry added a comment.Jun 26 2023, 6:29 AM
In T339896#8954960, @Pginer-WMF wrote:

@KartikMistry, an adjustment in the configuration may be needed for the following languages:

  • Dogri (doi/doi_Deva)
  • Bodo (brx/brx_Deva)
  • Goan (gom/gom_Deva)

For these languages the configuration should capture only the language pairs involving English (not all combinations between them). The reason is that IndicTrans2 only supports translations from/to English and NLLB-200 (which supports all combinations) could not be used as a fallback for the above three languages since they are not supported by the model.

Thanks, @Pginer-WMF! This is fixed and deployed now.

Pginer-WMF added a comment.Jun 26 2023, 11:45 AM

Translation from English to Goan seems to be failing.
Going to https://translate.wmcloud.org/ and translaitng form English to "gom" results in perpetual loading state.
Trying in Section Translation shows MT not available error:

gom.m.wikipedia.org_w_index.php_title=Special_ContentTranslation&from=en&to=gom&page=Malvani%20cuisine&sx=true(iPhone SE).png (1×750 px, 170 KB)

JS Console shows:

cxserver.wikimedia.org/v2/translate/en/gom/MinT:1 Failed to load resource: the server responded with a status of 500 ()

gerritbot added a comment.Jun 26 2023, 1:11 PM

Change 933095 had a related patch set uploaded (by KartikMistry; author: KartikMistry):

[mediawiki/services/machinetranslation@master] Add gom code to WIKI2NLLBCODES

https://gerrit.wikimedia.org/r/933095

gerritbot added a project: Patch-For-Review.Jun 26 2023, 1:12 PM
KartikMistry added a comment.Jun 26 2023, 1:12 PM
In T339896#8963672, @Pginer-WMF wrote:

Translation from English to Goan seems to be failing.
Going to https://translate.wmcloud.org/ and translaitng form English to "gom" results in perpetual loading state.
Trying in Section Translation shows MT not available error:

gom.m.wikipedia.org_w_index.php_title=Special_ContentTranslation&from=en&to=gom&page=Malvani%20cuisine&sx=true(iPhone SE).png (1×750 px, 170 KB)

JS Console shows:

cxserver.wikimedia.org/v2/translate/en/gom/MinT:1 Failed to load resource: the server responded with a status of 500 ()

Thanks for reporting, Pau! Submitted fix. It should be deployed tomorrow.

KartikMistry moved this task from In Progress to In Review on the Language-Team (Language-2023-April-June) board.Jun 27 2023, 4:06 AM
gerritbot added a comment.Jun 27 2023, 5:36 AM

Change 933095 merged by jenkins-bot:

[mediawiki/services/machinetranslation@master] Add gom code to WIKI2NLLBCODES

https://gerrit.wikimedia.org/r/933095

gerritbot added a comment.Jun 27 2023, 5:59 AM

Change 933221 had a related patch set uploaded (by KartikMistry; author: KartikMistry):

[operations/deployment-charts@master] Update MinT to 2023-06-27-053706-production

https://gerrit.wikimedia.org/r/933221

gerritbot added a comment.Jun 27 2023, 8:41 AM

Change 933221 merged by jenkins-bot:

[operations/deployment-charts@master] Update MinT to 2023-06-27-053706-production

https://gerrit.wikimedia.org/r/933221

Maintenance_bot removed a project: Patch-For-Review.Jun 27 2023, 9:11 AM
Stashbot added a comment.Jun 27 2023, 9:11 AM

Mentioned in SAL (#wikimedia-operations) [2023-06-27T09:11:33Z] <kart_> Updated MinT to 2023-06-27-053706-production (T339896, T340236)

KartikMistry added a comment.Jun 27 2023, 9:12 AM
In T339896#8963903, @KartikMistry wrote:
In T339896#8963672, @Pginer-WMF wrote:

Translation from English to Goan seems to be failing.
Going to https://translate.wmcloud.org/ and translaitng form English to "gom" results in perpetual loading state.
Trying in Section Translation shows MT not available error:

gom.m.wikipedia.org_w_index.php_title=Special_ContentTranslation&from=en&to=gom&page=Malvani%20cuisine&sx=true(iPhone SE).png (1×750 px, 170 KB)

JS Console shows:

cxserver.wikimedia.org/v2/translate/en/gom/MinT:1 Failed to load resource: the server responded with a status of 500 ()

Thanks for reporting, Pau! Submitted fix. It should be deployed tomorrow.

Fixed and deployed.

Pginer-WMF edited projects, added Language-Team (Language-2023-July-September); removed Language-Team (Language-2023-April-June).Jun 30 2023, 11:45 AM
Pginer-WMF moved this task from Quarter Backlog to In Review on the Language-Team (Language-2023-July-September) board.
KartikMistry added a comment.Jul 3 2023, 6:22 AM

@UOzurumba We can notify communities about the enablement of MinT now.

UOzurumba updated the task description. (Show Details)Jul 3 2023, 8:52 AM
KartikMistry moved this task from In Review to Done on the Language-Team (Language-2023-July-September) board.Jul 3 2023, 11:42 AM
UOzurumba added a comment.Jul 4 2023, 11:17 PM
In T339896#8984452, @KartikMistry wrote:

@UOzurumba We can notify communities about the enablement of MinT now.

@KartikMistry Done. Thanks!

Pginer-WMF closed this task as Resolved.Jul 5 2023, 7:46 AM
Pginer-WMF mentioned this in T337656: Explore using IndicTrans2 - better model supporting 22 Indic languages.Jul 21 2023, 9:55 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits