Improve MinT punctuation support for Japanese
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Pginer-WMF
	Oct 23 2023, 10:04 AM

Description

As reported here Japanese language doesn't normally apply punctuation marks as , or ., but either 、 or 。. However, MinT seems to be generating Japanese sentences with latin-based punctuation (note the presence of , and . below):

このページは,翻訳と反省を提供できる [[$2のウィキメディア財団統治ウィキ]]の"$1"に移動されました.

This ticket proposes to analyze the current situation and explore options to improve the punctuation support, including post-processing of the results as it was done for Punjabi (T215083).

Details

	Subject	Repo	Branch	Lines +/-
	Update MinT to 2024-03-26-120044-production	operations/deployment-charts	master	+1 -1
	Normalize japanese punctuations	mediawiki/services/machinetranslation	master	+5 -0

Customize query in gerrit

Related Objects

Mentioned In: T347930: Odia Language Translation Number not translating
T355304: Enable Softcatalà models for more language pairs in MinT test instance
Mentioned Here: T347930: Odia Language Translation Number not translating
T355304: Enable Softcatalà models for more language pairs in MinT test instance
T215083: Change "." to "।" in Content Translation into Punjabi Wikipedia

Event Timeline

Pginer-WMF created this task.Oct 23 2023, 10:04 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 23 2023, 10:04 AM

Pginer-WMF triaged this task as Medium priority.Oct 23 2023, 10:04 AM

Pginer-WMF moved this task from Backlog to General translation functionality on the MinT board.Oct 23 2023, 11:42 AM

Lemonaka subscribed.Oct 29 2023, 12:28 PM

Pginer-WMF moved this task from Quarter Backlog to Priority: Translation on the Language-Team (Language-2023-October-December) board.Nov 2 2023, 4:17 PM

Pginer-WMF edited projects, added Language-Team (Language-2024-January-March); removed Language-Team (Language-2023-October-December).Jan 8 2024, 10:39 AM

Pginer-WMF moved this task from Quarter Backlog to Priority: Translation on the Language-Team (Language-2024-January-March) board.

Change #1014151 had a related patch set uploaded (by Santhosh; author: Santhosh):

[mediawiki/services/machinetranslation@master] Normalize japanese punctuations

https://gerrit.wikimedia.org/r/1014151

santhosh claimed this task.Mar 26 2024, 5:25 AM

gerritbot added a project: Patch-For-Review.Mar 26 2024, 5:25 AM

santhosh moved this task from Priority: Translation to In Review on the Language-Team (Language-2024-January-March) board.Mar 26 2024, 5:25 AM

Change #1014151 merged by jenkins-bot:

[mediawiki/services/machinetranslation@master] Normalize japanese punctuations

https://gerrit.wikimedia.org/r/1014151

Change #1014729 had a related patch set uploaded (by KartikMistry; author: KartikMistry):

[operations/deployment-charts@master] Update MinT to 2024-03-26-120044-production

https://gerrit.wikimedia.org/r/1014729

Change #1014729 merged by jenkins-bot:

[operations/deployment-charts@master] Update MinT to 2024-03-26-120044-production

https://gerrit.wikimedia.org/r/1014729

Mentioned in SAL (#wikimedia-operations) [2024-03-27T07:11:01Z] <kart_> Updated MinT to 2024-03-26-120044-production (T347930, T355304, T349487)

santhosh moved this task from In Review to Needs QA on the Language-Team (Language-2024-January-March) board.Mar 28 2024, 5:26 AM

Nikerabbit removed a project: Patch-For-Review.Apr 2 2024, 11:18 AM

Pginer-WMF edited projects, added Language-Team (Language-2024-April-June); removed Language-Team (Language-2024-January-March).Apr 2 2024, 3:19 PM

Pginer-WMF moved this task from Quarter Backlog to Needs QA on the Language-Team (Language-2024-April-June) board.

Punctuation marks seem to be using the correct characters when translating into Japanese now. I tried two text samples from English and Catalan in the test instance and only the expected symbols 、 or 。 appear, with no trace of the , or . symbols from the original texts that were incorrectly appearing in the translation before the fix: