Page MenuHomePhabricator
Create Task
Maniphest T336525

Review code mappings for MinT
Closed, ResolvedPublic
Actions

Description

Language codes are used to identify languages for Wikipedia and for the translation models used in MinT. Since there are different standards for language codes (using 2 or 3 letters) and exceptional cases, there is code in the system to map the language codes. This task will identify instances that may need fixing.

Mapping from Wikipedia codes to NLLB-200 codes

Mapping in this file may need revision on some Wikipedia codes mapping to the NLLB-200 languages:

  • "fil": "tgl_Latn". Expected: "tl": "tgl_Latn" Tagalog Wikipedia uses the "tl" code.
  • "gaz": "gaz_Latn". Expected "om": "gaz_Latn" Oromo Wikipedia uses the "om" code. The issue is only present in the "nllb200-600M"model. The "nllb-wikipedia" uses the "om" code as expected, making MinT work currently for the language.
  • "npi": "npi_Deva". Expected "ne": "npi_Deva". Nepali Wikipedia uses "ne" code.
  • "swh": "swh_Latn". Expected "sw": "swh_Latn". Swahili Wikipedia uses "sw" code.
  • "als": "als_Latn". Expected "sq": "als_Latn". NLLB-200 uses "als" for Tosk Albanian. The Albanian Wikipedia (for all Albaniant variants?) uses "sq". Note that Wikipedia uses the non-standard "als" code for Swiss German which is unrelated to Tosk Albanian.
  • "plt": "plt_Latn". Expected: "mg": "plt_Latn". The Malagasy Wikipedia uses "mg" code.
  • "zsm": "zsm_Latn". Expected: "ms": "zsm_Latn". The Malay wikipedia uses "ms" code.
  • "ory": "ory_Orya". Expected "or": "ory_Orya". Oriya Wikipedia uses "or" code.
  • "fuv": "fuv_Latn". Expected "ff": "fuv_Latn". Fulfulde Wikipedia uses "ff" code.
  • "dik": "dik_Latn". Expected "din": "dik_Latn". Dinka Wikipedia uses "din" code.
  • "azj": "azj_Latn". Expected "az": "azj_Latn". Azerbaijani Wikipedia uses "az" code.
  • "bho": "bho_Deva". Expected "bh": "bho_Deva". Bhojpuri Wikipedia uses "bh" code.
  • "kmr": "kmr_Latn". Expected "ku": "kmr_Latn". Kurdish Wikipedia uses "ku" code.
  • "lvs": "lvs_Latn". Expected "lv": "lvs_Latn". Latvian Wikipedia uses "lv" code.
  • "khk": "khk_Cyrl". Expected "mn": "khk_Cyrl". Mongolian Wikipedia uses "mn" code.
  • "pes": "pes_Arab". Expected "fa": "pes_Arab". Persian Wikipedia uses "fa" code.
  • "uzn": "uzn_Latn". Expected "uz": "uzn_Latn". Uzbek Wikipedia uses "uz" code.
  • "ydd": "ydd_Hebr". Expected "yi": "ydd_Hebr". Yiddish Wikipedia uses "yi" code.
  • "pbt": "pbt_Arab". Expected "ps": "pbt_Arab". Pashto Wikipedia uses "ps" code.
  • "quy": "quy_Latn". Expected "qu": "quy_Latn". Quechua Wikipedia uses "qu" code.
  • "ayr": "ayr_Latn". Expected "ay": "ayr_Latn" Aymara Wikipedia uses the "ay" code. The issue is only present in the "nllb200-600M"model. The "nllb-wikipedia" uses the "ay" code as expected, making MinT work currently for the language.
  • Add: "arz": "arz_Arab" since support for Egyptian Arabic was missing.
  • Add: "ary": "ary_Arab" since support for Moroccan Arabic was missing.

List of supported languages

The list of supported languages in this file is expected to use the Wikipedia language codes. It may need to adjust the following:

  • bjn is duplicated.
  • gaz. Expected: om
  • npi. Expected: ne
  • fil. Expected: tl
  • swh. Expected: sw
  • als. Expected sq
  • plt. Expected: mg
  • zsm. Expected: ms
  • ory. Expected or
  • fuv. Expected ff
  • dik. Expected din
  • azj. Expected az
  • bho. Expected bh
  • kmr. Expected ku
  • lvs. Expected lv
  • khk. Expected mn
  • pes. Expected fa
  • uzn. Expected uz
  • ydd. Expected yi
  • pbt. Expected ps
  • quy. Expected qu
  • ayr". Expected ay
  • Add: arz under nllb200-600M since support for Egyptian Arabic was missing.
  • Add: ary under nllb200-600M since support for Moroccan Arabic was missing.

Details

SubjectRepoBranchLines +/-
MinT: Update to 2023-06-01-041041-productionoperations/deployment-chartsmaster+1 -1
config.yaml: Fix language code for Mongolianmediawiki/services/machinetranslationmaster+1 -1
Updated MinT to 2023-05-16-112045-productionoperations/deployment-chartsmaster+1 -1
Update language code mappingsmediawiki/services/machinetranslationmaster+143 -145
Customize query in gerrit

Related Objects

Event Timeline

Pginer-WMF created this task.May 11 2023, 4:36 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 11 2023, 4:36 PM
Pginer-WMF triaged this task as Medium priority.May 11 2023, 4:37 PM
Amire80 subscribed.May 11 2023, 5:23 PM

Maybe also fix ory and or. They both refer to Odia. For some reason, we seem to have both in NLLB. Wikimedia uses or, and it should be used as much as possible.

Pginer-WMF updated the task description. (Show Details)May 11 2023, 5:49 PM
Pginer-WMF added a comment.May 12 2023, 7:22 AM
In T336525#8845585, @Amire80 wrote:

Maybe also fix ory and or. They both refer to Odia. For some reason, we seem to have both in NLLB. Wikimedia uses or, and it should be used as much as possible.

Thanks Amir! I added items to adjust Odia too.

santhosh subscribed.May 15 2023, 4:21 AM
KartikMistry claimed this task.May 15 2023, 11:54 AM
gerritbot added a comment.May 15 2023, 11:54 AM

Change 919828 had a related patch set uploaded (by KartikMistry; author: KartikMistry):

[mediawiki/services/machinetranslation@master] Update code mappings

https://gerrit.wikimedia.org/r/919828

gerritbot added a project: Patch-For-Review.May 15 2023, 11:54 AM
KartikMistry moved this task from Quarter Backlog to In Review on the Language-Team (Language-2023-April-June) board.May 15 2023, 11:54 AM
Pginer-WMF updated the task description. (Show Details)May 15 2023, 1:38 PM
Pginer-WMF updated the task description. (Show Details)May 15 2023, 2:07 PM
Pginer-WMF updated the task description. (Show Details)May 15 2023, 2:12 PM
Pginer-WMF updated the task description. (Show Details)May 15 2023, 2:19 PM
Pginer-WMF updated the task description. (Show Details)May 15 2023, 2:25 PM
Pginer-WMF updated the task description. (Show Details)May 15 2023, 2:29 PM
Pginer-WMF updated the task description. (Show Details)May 15 2023, 2:34 PM
Pginer-WMF updated the task description. (Show Details)May 15 2023, 2:37 PM
Pginer-WMF added a comment.May 15 2023, 2:42 PM

For Norwegian Bokmål, the configuration uses "nb" as code, but Wikipedia redirects "nb" to "no". Based on previous issues with Norwegian variants, would be worth checking which should be the best code to use here.

Pginer-WMF updated the task description. (Show Details)May 15 2023, 2:44 PM
Pginer-WMF updated the task description. (Show Details)May 15 2023, 2:48 PM
Restricted Application added a subscriber: Huji. · View Herald TranscriptMay 15 2023, 2:48 PM
Pginer-WMF updated the task description. (Show Details)May 15 2023, 2:57 PM
Pginer-WMF updated the task description. (Show Details)May 15 2023, 2:59 PM
Pginer-WMF updated the task description. (Show Details)May 15 2023, 3:06 PM
Pginer-WMF updated the task description. (Show Details)
Pginer-WMF updated the task description. (Show Details)May 15 2023, 3:09 PM
KartikMistry updated the task description. (Show Details)May 16 2023, 4:14 AM
gerritbot added a comment.May 16 2023, 4:42 AM

Change 919828 merged by jenkins-bot:

[mediawiki/services/machinetranslation@master] Update language code mappings

https://gerrit.wikimedia.org/r/919828

Maintenance_bot removed a project: Patch-For-Review.May 16 2023, 5:10 AM
Pginer-WMF moved this task from Backlog to Product integration on the MinT board.May 16 2023, 8:53 AM
gerritbot added a comment.May 16 2023, 11:34 AM

Change 920250 had a related patch set uploaded (by KartikMistry; author: KartikMistry):

[operations/deployment-charts@master] Updated MinT to 2023-05-16-112045-production

https://gerrit.wikimedia.org/r/920250

gerritbot added a project: Patch-For-Review.May 16 2023, 11:34 AM
gerritbot added a comment.May 16 2023, 12:02 PM

Change 920250 merged by jenkins-bot:

[operations/deployment-charts@master] Updated MinT to 2023-05-16-112045-production

https://gerrit.wikimedia.org/r/920250

Maintenance_bot removed a project: Patch-For-Review.May 16 2023, 12:12 PM
KartikMistry moved this task from In Review to Done on the Language-Team (Language-2023-April-June) board.May 16 2023, 12:21 PM
KartikMistry updated the task description. (Show Details)May 16 2023, 12:28 PM
Pginer-WMF added a comment.May 31 2023, 2:44 PM

@KartikMistry I think the following change was not actually made:

List of supported languages

The list of supported languages in this file is expected to use the Wikipedia language codes. It may need to adjust the following:
[...]

  • khk. Expected mn

Mongolian is expected to be listed there using the Wikipedia code (mn) instead of the NLLB-200 code (khk)

gerritbot added a comment.May 31 2023, 2:55 PM

Change 924966 had a related patch set uploaded (by KartikMistry; author: KartikMistry):

[mediawiki/services/machinetranslation@master] config.yaml: Fix language code for Mongolian

https://gerrit.wikimedia.org/r/924966

gerritbot added a project: Patch-For-Review.May 31 2023, 2:55 PM
KartikMistry moved this task from Done to In Review on the Language-Team (Language-2023-April-June) board.May 31 2023, 2:55 PM
gerritbot added a comment.Jun 1 2023, 4:10 AM

Change 924966 merged by jenkins-bot:

[mediawiki/services/machinetranslation@master] config.yaml: Fix language code for Mongolian

https://gerrit.wikimedia.org/r/924966

Maintenance_bot removed a project: Patch-For-Review.Jun 1 2023, 4:10 AM
gerritbot added a comment.Jun 1 2023, 4:45 AM

Change 924915 had a related patch set uploaded (by KartikMistry; author: KartikMistry):

[operations/deployment-charts@master] MinT: Update to 2023-06-01-041041-production

https://gerrit.wikimedia.org/r/924915

gerritbot added a project: Patch-For-Review.Jun 1 2023, 4:45 AM
gerritbot added a comment.Jun 1 2023, 5:36 AM

Change 924915 merged by jenkins-bot:

[operations/deployment-charts@master] MinT: Update to 2023-06-01-041041-production

https://gerrit.wikimedia.org/r/924915

Maintenance_bot removed a project: Patch-For-Review.Jun 1 2023, 6:10 AM
Stashbot added a comment.Jun 1 2023, 6:16 AM

Mentioned in SAL (#wikimedia-operations) [2023-06-01T06:16:00Z] <kart_> Updated MinT to 2023-06-01-041041-production (T336525)

KartikMistry added a comment.Jun 1 2023, 6:16 AM
In T336525#8892746, @Pginer-WMF wrote:

@KartikMistry I think the following change was not actually made:

List of supported languages

The list of supported languages in this file is expected to use the Wikipedia language codes. It may need to adjust the following:
[...]

  • khk. Expected mn

Mongolian is expected to be listed there using the Wikipedia code (mn) instead of the NLLB-200 code (khk)

Thanks, Pau for noticing this. Fixed and deployed.

KartikMistry moved this task from In Review to Done on the Language-Team (Language-2023-April-June) board.Jun 1 2023, 6:19 AM
Pginer-WMF closed this task as Resolved.Jun 1 2023, 7:16 AM
Pginer-WMF mentioned this in T337910: Adjust Norwegian mapping for MinT configuration.Jun 1 2023, 8:12 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits