Enable MT based on closely-related languages based on community input
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Pginer-WMF
	Jul 27 2020, 9:15 AM

Description

Some languages lacking Machine Translation (MT) support may benefit from having access to the MT available for closely related languages. It may be preferred to correct the language differences rather than starting from scratch.

Based on input from their communities, this ticket proposes to enable MT for the following cases:

On Serbo-Croatian (sh) Wikipedia enable Bosnian (bs) MT with Google and Yandex (details)

On Wu (wuu) Wikipedia enable Chinese (zh) MT with Google, Yandex, LingoCloud and Youdao (T199523)

On Cantonese (zh-yue) Wikipedia enable the Traditional script variant of Chinese MT which is only available with Google (using code zh-TW).

On Gan (gan) Wikipedia enable the Traditional script variant of Chinese MT which is only available with Google (using code zh-TW).

On Belarusian Taraškievica (be-tarask) Enable Belarusian (be) MT with Google and Yandex.

These changes are similar to the way Simple English Wikipedia (T196354) had English MT is exposed.

Details

Subject	Repo	Branch	Lines +/-
Update cxserver to 2020-08-05-070016-production	operations/deployment-charts	master	+3 -3
Enable MT based on closely-related languages (be-tarask)	mediawiki/services/cxserver	master	+1 -4
Enable MT based on closely-related languages (gan and zh-yue)	mediawiki/services/cxserver	master	+5 -1
Enable MT based on closely-related languages (wuu)	mediawiki/services/cxserver	master	+23 -4
Enable MT based on closely-related languages (sh)	mediawiki/services/cxserver	master	+6 -2

Customize query in gerrit

Related Objects

Mentioned In: T304865: Enable Content and Section Translation for Cantonese Wikipedia
T343450: Enable MinT for closely-related languages based on community input
T343340: Identify most useful languages for future OpusMT models
T309173: Enable newly supported languages by Flores in the test instance
T256194: Provide section order information in the section suggestions API
T199523: Expose Machine Translation services supporting Chinese to closer languages/variants
T257943: Create Wikipedia Kotava
Mentioned Here: T256194: Provide section order information in the section suggestions API
T257943: Create Wikipedia Kotava
T196354: Enable MT for "Simple English" as if it was (regular) English
T199523: Expose Machine Translation services supporting Chinese to closer languages/variants

Event Timeline

Pginer-WMF created this task.Jul 27 2020, 9:15 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 27 2020, 9:15 AM

Pginer-WMF triaged this task as Medium priority.Jul 27 2020, 9:15 AM

Pginer-WMF updated the task description. (Show Details)

Pginer-WMF updated the task description. (Show Details)Jul 27 2020, 9:21 AM

Pginer-WMF assigned this task to KartikMistry.Jul 29 2020, 7:57 AM

KartikMistry updated the task description. (Show Details)Jul 30 2020, 5:26 AM

Pginer-WMF edited projects, added Language-Team (Language-2020-July-September); removed Language-Team (Language-2020-Focus-Sprint).Aug 3 2020, 7:57 AM

Change 618009 had a related patch set uploaded (by KartikMistry; owner: KartikMistry):
[mediawiki/services/cxserver@master] Enable MT based on closely-related languages (sh)

https://gerrit.wikimedia.org/r/618009

gerritbot added a project: Patch-For-Review.Aug 3 2020, 8:49 AM

Pginer-WMF moved this task from Quarter Backlog to Priority Backlog on the Language-Team (Language-2020-July-September) board.Aug 3 2020, 9:18 AM

KartikMistry moved this task from Priority Backlog to In Progress on the Language-Team (Language-2020-July-September) board.Aug 3 2020, 11:54 AM

Change 618050 had a related patch set uploaded (by KartikMistry; owner: KartikMistry):
[mediawiki/services/cxserver@master] Enable MT based on closely-related languages (wuu)

https://gerrit.wikimedia.org/r/618050

Change 618176 had a related patch set uploaded (by KartikMistry; owner: KartikMistry):
[mediawiki/services/cxserver@master] Enable MT based on closely-related languages (gan and zh-yue)

https://gerrit.wikimedia.org/r/618176

Change 618228 had a related patch set uploaded (by KartikMistry; owner: KartikMistry):
[mediawiki/services/cxserver@master] Enable MT based on closely-related languages based on community input (be-tarask)

https://gerrit.wikimedia.org/r/618228

KartikMistry moved this task from In Progress to In Review on the Language-Team (Language-2020-July-September) board.Aug 4 2020, 6:40 AM

Change 618009 merged by jenkins-bot:
[mediawiki/services/cxserver@master] Enable MT based on closely-related languages (sh)

https://gerrit.wikimedia.org/r/618009

Change 618050 merged by KartikMistry:
[mediawiki/services/cxserver@master] Enable MT based on closely-related languages (wuu)

https://gerrit.wikimedia.org/r/618050

Change 618176 merged by jenkins-bot:
[mediawiki/services/cxserver@master] Enable MT based on closely-related languages (gan and zh-yue)

https://gerrit.wikimedia.org/r/618176

Change 618228 merged by jenkins-bot:
[mediawiki/services/cxserver@master] Enable MT based on closely-related languages (be-tarask)

https://gerrit.wikimedia.org/r/618228

KartikMistry moved this task from In Review to Recheck after deployment on the Language-Team (Language-2020-July-September) board.Aug 5 2020, 7:01 AM

Maintenance_bot removed a project: Patch-For-Review.Aug 5 2020, 7:10 AM

Change 618525 had a related patch set uploaded (by KartikMistry; owner: KartikMistry):
[operations/deployment-charts@master] Update cxserver to 2020-08-05-070016-production

https://gerrit.wikimedia.org/r/618525

gerritbot added a project: Patch-For-Review.Aug 5 2020, 12:25 PM

Change 618525 merged by jenkins-bot:
[operations/deployment-charts@master] Update cxserver to 2020-08-05-070016-production

https://gerrit.wikimedia.org/r/618525

Maintenance_bot removed a project: Patch-For-Review.Aug 6 2020, 6:10 AM

Mentioned in SAL (#wikimedia-operations) [2020-08-06T12:06:33Z] <kart_> Updated cxserver to 2020-08-05-070016-production (T258919, T199523, T257943, T256194)

Stashbot mentioned this in T257943: Create Wikipedia Kotava.Aug 6 2020, 12:06 PM

Stashbot mentioned this in T199523: Expose Machine Translation services supporting Chinese to closer languages/variants.

Stashbot mentioned this in T256194: Provide section order information in the section suggestions API.

KartikMistry updated the task description. (Show Details)Aug 6 2020, 12:08 PM

KartikMistry moved this task from Recheck after deployment to Needs QA on the Language-Team (Language-2020-July-September) board.

For serbo-croatian, Apertium is available to use for MT but the translation ends up like this

Maybe we should remove Apertium?

Also, I can't assure that this change is working as supposed because I don't speak the languages used so I don't know which language is the output of the MT

In T258919#6370201, @Jpita wrote:

For serbo-croatian, Apertium is available to use for MT but the translation ends up like this
Maybe we should remove Apertium?

We can keep it. No need to remove it unless specifically asked by the community.

Also, I can't assure that this change is working as supposed because I don't speak the languages used so I don't know which language is the output of the MT

We deployed this for Google. Did that worked well? Only "MT is working" is sufficient to check as of now.

KartikMistry moved this task from Pending input for QA to Needs QA on the Language-Team (Language-2020-July-September) board.Aug 10 2020, 8:11 AM

In T258919#6370201, @Jpita wrote:

Also, I can't assure that this change is working as supposed because I don't speak the languages used so I don't know which language is the output of the MT

Some ways that may be useful for checking this:

Compare with the alternative language in Content Translation. For example, you can translate an article from English to Serbo-Croatian. Then translate the same article from English to Bosnian. Since Bosian MT is expected to be used in both cases the MT provided should be the same.
Use the MT service website. For example, you can translate an article from English to Serbo-Croatian using Google. then go to the Google Translate website and paste the english text to get the Bosnian translation, which should be the same as you got in Content Translation.

The above comparisons can be made by looking at the texts side by side, but for scripts you may be unfamiliar with an online diff tool may be useful.

In T258919#6371934, @Pginer-WMF wrote:

In T258919#6370201, @Jpita wrote:

Also, I can't assure that this change is working as supposed because I don't speak the languages used so I don't know which language is the output of the MT

Some ways that may be useful for checking this:

Compare with the alternative language in Content Translation. For example, you can translate an article from English to Serbo-Croatian. Then translate the same article from English to Bosnian. Since Bosian MT is expected to be used in both cases the MT provided should be the same.

Use the MT service website. For example, you can translate an article from English to Serbo-Croatian using Google. then go to the Google Translate website and paste the english text to get the Bosnian translation, which should be the same as you got in Content Translation.

The above comparisons can be made by looking at the texts side by side, but for scripts you may be unfamiliar with an online diff tool may be useful.