Long sequence of a repeated word appears only when using MinT but not NLLB-200 directly
Open, MediumPublic
Actions

Assigned To

Authored By

	Pginer-WMF
	Thu, May 11, 10:59 AM

Description

When translating the article "Paneer" from English to Aymara (quick link) the first text paragraph is translated differently for the "NLLB-200" and "MinT" options. In particular, the MinT option produces a translation where the word "quesito" is repeated many times.

A video and copied contents below:

Original text	NLLB-200	MinT
Paneer (pronounced [/pəˈniːr/]), also known as ponir (pronounced [po̯ni̯r]), is a fresh acid-set cheese common in the Indian subcontinent (India, Bangladesh, Bhutan, Maldives, Nepal, Pakistan and Sri Lanka) made from full-fat buffalo milk or cow milk. It is a non-aged, non-melting soft cheese made by curdling milk with a fruit- or vegetable-derived acid, such as lemon juice.	Paneer (pronunciado [/pəˈniːr/], ponir (pronunciáu [po̯ni̯r]), ukax mä quesito quesito acido ukampiw k'achat k'achata, India, Bangladesh, Bhutan, Maldives, Nepal, Pakistán ukat Sri Lanka) uksanakanx wali uñt'ata. Uka quesosti janiw ch'uxñatatkiti, janiw ch'uqiyatäkiti, lechex ch'uqt'atawa, jan ukax ch'uqinakat ch'uñatatatawa, leche ukamp ch'uyanchatawa.	Paneer (pronunciado [/pəˈniːr/], ponir (pronunciado [po̯ni̯r]), ukax mä quesito quesito quesito quesito quesito quesito quesito quesito quesito quesito quesito quesito quesito quesito quesito quesito quesito quesito quesito quesito quesito quesito quesito quesito quesito quesito quesito quesito quesito quesito quesito quesito quesito quesito quesito quesito quesito quesito quesito quesito quesito quesito quesito quesito quesito quesito quesito quesito quesito quesito quesito quesito quesito quesito quesito quesito quesito quesito quesito quesito quesito quesito quesito quesito quesito quesito quesito quesito quesito quesito quesito quesito quesito quesito qu Uka quesosti janiw sinti ch'uxñatatkiti, janiw ch'uxñatatkiti, lechex achunakata jan ukax ch'uxña quqanakat luratawa, limón quqanjama.

Details

Project	Branch	Lines +/-	Subject
operations/deployment-charts	master	+5 -5	MinT: Update to 2023-05-18-060931-production and Set CT2_INTRA_THREADS to 0
mediawiki/services/machinetranslation	master	+1 -1	Use the correct setting for avoiding ngram repitition
operations/deployment-charts	master	+1 -1	Updated MinT to 2023-05-16-112045-production
mediawiki/services/machinetranslation	master	+2 -0	Set repetition_penalty for translator

Customize query in gerrit

Event Timeline

Pginer-WMF created this task.Thu, May 11, 10:59 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptThu, May 11, 10:59 AM

Pginer-WMF triaged this task as Medium priority.Thu, May 11, 11:03 AM

Change 919042 had a related patch set uploaded (by Santhosh; author: Santhosh):

[mediawiki/services/machinetranslation@master] Set repetition_penalty for translator

https://gerrit.wikimedia.org/r/919042

gerritbot added a project: Patch-For-Review.Thu, May 11, 11:45 AM

Pginer-WMF moved this task from Backlog to General translation functionality on the MinT board.Thu, May 11, 3:18 PM

santhosh claimed this task.Fri, May 12, 3:54 AM

santhosh moved this task from Quarter Backlog to In Review on the Language-Team (Language-2023-April-June) board.

Change 919042 merged by jenkins-bot:

[mediawiki/services/machinetranslation@master] Set repetition_penalty for translator

https://gerrit.wikimedia.org/r/919042

Maintenance_bot removed a project: Patch-For-Review.Mon, May 15, 6:10 AM

Change 920250 had a related patch set uploaded (by KartikMistry; author: KartikMistry):

[operations/deployment-charts@master] Updated MinT to 2023-05-16-112045-production

https://gerrit.wikimedia.org/r/920250

gerritbot added a project: Patch-For-Review.Tue, May 16, 11:34 AM

Change 920250 merged by jenkins-bot:

[operations/deployment-charts@master] Updated MinT to 2023-05-16-112045-production

https://gerrit.wikimedia.org/r/920250

Maintenance_bot removed a project: Patch-For-Review.Tue, May 16, 12:13 PM

This should be fix with latest deployment, but I'm keeping it in QA as of now.

The issue is fixed indeed. Interestingly, the MinT translation has only one instance of "quesito" while the translation from NLLB-200 presents a repetition ("quesito quesito"). On the other hand, from the sentence "Paneer (pronounced [/pəˈniːr/]), also known as ponir (pronounced [po̯ni̯r])", the MinT translation seems to skip the second part (maybe because the repetition penalty?), note the IPA and the parenthesis as well as the single occurrence of "pronunciado" ("pronounced").

Original text	NLLB-200	MinT
Paneer (pronounced [/pəˈniːr/]), also known as ponir (pronounced [po̯ni̯r]), is a fresh acid-set cheese common in the Indian subcontinent (India, Bangladesh, Bhutan, Maldives, Nepal, Pakistan and Sri Lanka) made from full-fat buffalo milk or cow milk. It is a non-aged, non-melting soft cheese made by curdling milk with a fruit- or vegetable-derived acid, such as lemon juice.	Paneer (pronunciado [/pəˈniːr/], ponir (pronunciáu [po̯ni̯r]), ukax mä quesito quesito acido ukampiw k'achat k'achata, India, Bangladesh, Bhutan, Maldives, Nepal, Pakistán ukat Sri Lanka) uksanakanx wali uñt'ata. Uka quesosti janiw ch'uxñatatkiti, janiw ch'uqiyatäkiti, lechex ch'uqt'atawa, jan ukax ch'uqinakat ch'uñatatatawa, leche ukamp ch'uyanchatawa.	Paneer (pronunciado [/pəˈniːr /], ponir ukjamaraki uñt'ata, ukax mä quesito acido-samp k'achat ukhamawa subcontinente indiano ukanxa (India, Bangladesh, Bhutan, Maldives, Nepal, Pakistán ukat Sri Lanka) wali suma lurt'atawa. Uka quesosti janiw sinti ch'uxñatatkiti, jan ukasti lechex achunakata ukat k'allk'u manqhat luratawa.

Change 920813 had a related patch set uploaded (by Santhosh; author: Santhosh):

[mediawiki/services/machinetranslation@master] Use the correct setting for avoiding ngram repitition

https://gerrit.wikimedia.org/r/920813

gerritbot added a project: Patch-For-Review.Thu, May 18, 5:49 AM

Change 920813 merged by jenkins-bot:

[mediawiki/services/machinetranslation@master] Use the correct setting for avoiding ngram repitition

https://gerrit.wikimedia.org/r/920813

Maintenance_bot removed a project: Patch-For-Review.Thu, May 18, 6:10 AM

Change 920687 had a related patch set uploaded (by KartikMistry; author: KartikMistry):

[operations/deployment-charts@master] MinT: Update to 2023-05-18-060931-production and Set CT2_INTRA_THREADS to 0

https://gerrit.wikimedia.org/r/920687

gerritbot added a project: Patch-For-Review.Thu, May 18, 6:53 AM

Change 920687 merged by jenkins-bot:

[operations/deployment-charts@master] MinT: Update to 2023-05-18-060931-production and Set CT2_INTRA_THREADS to 0

https://gerrit.wikimedia.org/r/920687

Maintenance_bot removed a project: Patch-For-Review.Thu, May 18, 11:10 AM

Mentioned in SAL (#wikimedia-operations) [2023-05-18T11:36:45Z] <kart_> MinT: Update to 2023-05-18-060931-production and Set CT2_INTRA_THREADS to 0 (T336483)

santhosh@deploy1002:~$ curl 'https://machinetranslation.discovery.wmnet:30443/api/translate/en/ay' -X POST -H 'User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/112.0' -H 'Accept:*/*'  -H 'Accept-Encoding: gzip, deflate, br' -H 'Referer: https://translate.wmcloud.org/' -H 'Content-Type: application/json' -H 'Origin: https://translate.wmcloud.org' -H 'DNT: 1' -H 'Connection: keep-alive' -H 'Sec-Fetch-Dest: empty' -H 'Sec-Fetch-Mode: cors' -H 'Sec-Fetch-Site: same-origin' -H 'Pragma: no-cache' -H 'Cache-Control: no-cache' -H 'TE: trailers' --data-raw '{"text":"Paneer (pronounced [/pəˈniːr/]), also known as ponir (pronounced [po̯ni̯r]), is a fresh acid-set cheese common in the Indian subcontinent (India, Bangladesh, Bhutan, Maldives, Nepal, Pakistan and Sri Lanka) made from full-fat buffalo milk or cow milk. \n Itis a non-aged, non-melting soft cheese made by curdling milk with a fruit- or vegetable-derived acid, such as lemon juice."}'
`

{
"model":"nllb-wikiedia","
sourcelanguage":"en","
targetlanguage":"ay","
translation":"Paneer (pronunciado [/p\u0259\u02c8ni\u02d0r/], ponir (prononciado[po\u032fni\u032fr]), ukax m\u00e4 quesorakiwa, ukax subcontinente indiano (India, Bangladesh, Bhutan, Maldives, Nepal, Pakist\u00e1n ukat Sri Lanka) ukanx wali ch'ux\u00f1aki.\nUka quesosti janiw sinti ch'ux\u00f1atatkiti, janiw chhaqhayatakiti, lechex achunakata jan ukax ch 'ux\u00f1a quqanakat luratawa, lim\u00f3n jusi ukhama.",
"translationtime":1.1759014129638672
}

After the recent changes, the translation from MinT does no longer seem to have the previous issues of skipping parts of the translation. In fact, it seems to deal better than NLLB-200 with some aspects:

The parenthesis in "...Indian subcontinent (India, Bangladesh..." seems to get lost in the NLLB-200 translation, while it is preserved in MinT.
The repetition of words is no longer happening in MinT, while NLLB-200 still presents some instances ("quesito quesito").

Original text	NLLB-200	MinT
Paneer (pronounced [/pəˈniːr/]), also known as ponir (pronounced [po̯ni̯r]), is a fresh acid-set cheese common in the Indian subcontinent (India, Bangladesh, Bhutan, Maldives, Nepal, Pakistan and Sri Lanka) made from full-fat buffalo milk or cow milk. It is a non-aged, non-melting soft cheese made by curdling milk with a fruit- or vegetable-derived acid, such as lemon juice.	Paneer (pronunciado [/pəˈniːr/], ponir (pronunciáu [po̯ni̯r]), ukax mä quesito quesito acido ukampiw k'achat k'achata, India, Bangladesh, Bhutan, Maldives, Nepal, Pakistán ukat Sri Lanka) uksanakanx wali uñt'ata. Uka quesosti janiw ch'uxñatatkiti, janiw ch'uqiyatäkiti, lechex ch'uqt'atawa, jan ukax ch'uqinakat ch'uñatatatawa, leche ukamp ch'uyanchatawa.	Paneer (pronunciado [/pəˈniːr/], ponir (prononciado[po̯ni̯r]), ukax mä quesorakiwa, ukax subcontinente indiano (India, Bangladesh, Bhutan, Maldives, Nepal, Pakistán ukat Sri Lanka) ukanx wali ch'uxñaki. Uka quesosti janiw sinti ch'uxñatatkiti, janiw chhaqhayatakiti, lechex achunakata jan ukax ch 'uxña quqanakat luratawa, limón jusi ukhama.

	F36992112: Kapture 2023-05-11 at 12.01.17.webm
	Thu, May 11, 10:59 AM

Long sequence of a repeated word appears only when using MinT but not NLLB-200 directlyOpen, MediumPublicActions

Description

Details

Event Timeline

Long sequence of a repeated word appears only when using MinT but not NLLB-200 directly
Open, MediumPublic
Actions