Page MenuHomePhabricator

Long sequence of a repeated word appears only when using MinT but not NLLB-200 directly
Closed, ResolvedPublic

Description

When translating the article "Paneer" from English to Aymara (quick link) the first text paragraph is translated differently for the "NLLB-200" and "MinT" options. In particular, the MinT option produces a translation where the word "quesito" is repeated many times.

A video and copied contents below:

Original textNLLB-200MinT
Paneer (pronounced [/pəˈniːr/]), also known as ponir (pronounced [po̯ni̯r]), is a fresh acid-set cheese common in the Indian subcontinent (India, Bangladesh, Bhutan, Maldives, Nepal, Pakistan and Sri Lanka) made from full-fat buffalo milk or cow milk. It is a non-aged, non-melting soft cheese made by curdling milk with a fruit- or vegetable-derived acid, such as lemon juice.Paneer (pronunciado [/pəˈniːr/], ponir (pronunciáu [po̯ni̯r]), ukax mä quesito quesito acido ukampiw k'achat k'achata, India, Bangladesh, Bhutan, Maldives, Nepal, Pakistán ukat Sri Lanka) uksanakanx wali uñt'ata. Uka quesosti janiw ch'uxñatatkiti, janiw ch'uqiyatäkiti, lechex ch'uqt'atawa, jan ukax ch'uqinakat ch'uñatatatawa, leche ukamp ch'uyanchatawa.Paneer (pronunciado [/pəˈniːr/], ponir (pronunciado [po̯ni̯r]), ukax mä quesito quesito quesito quesito quesito quesito quesito quesito quesito quesito quesito quesito quesito quesito quesito quesito quesito quesito quesito quesito quesito quesito quesito quesito quesito quesito quesito quesito quesito quesito quesito quesito quesito quesito quesito quesito quesito quesito quesito quesito quesito quesito quesito quesito quesito quesito quesito quesito quesito quesito quesito quesito quesito quesito quesito quesito quesito quesito quesito quesito quesito quesito quesito quesito quesito quesito quesito quesito quesito quesito quesito quesito quesito quesito qu Uka quesosti janiw sinti ch'uxñatatkiti, janiw ch'uxñatatkiti, lechex achunakata jan ukax ch'uxña quqanakat luratawa, limón quqanjama.

Event Timeline

Change 919042 had a related patch set uploaded (by Santhosh; author: Santhosh):

[mediawiki/services/machinetranslation@master] Set repetition_penalty for translator

https://gerrit.wikimedia.org/r/919042

Change 919042 merged by jenkins-bot:

[mediawiki/services/machinetranslation@master] Set repetition_penalty for translator

https://gerrit.wikimedia.org/r/919042

Change 920250 had a related patch set uploaded (by KartikMistry; author: KartikMistry):

[operations/deployment-charts@master] Updated MinT to 2023-05-16-112045-production

https://gerrit.wikimedia.org/r/920250

Change 920250 merged by jenkins-bot:

[operations/deployment-charts@master] Updated MinT to 2023-05-16-112045-production

https://gerrit.wikimedia.org/r/920250

This should be fix with latest deployment, but I'm keeping it in QA as of now.

The issue is fixed indeed. Interestingly, the MinT translation has only one instance of "quesito" while the translation from NLLB-200 presents a repetition ("quesito quesito"). On the other hand, from the sentence "Paneer (pronounced [/pəˈniːr/]), also known as ponir (pronounced [po̯ni̯r])", the MinT translation seems to skip the second part (maybe because the repetition penalty?), note the IPA and the parenthesis as well as the single occurrence of "pronunciado" ("pronounced").

Original textNLLB-200MinT
Paneer (pronounced [/pəˈniːr/]), also known as ponir (pronounced [po̯ni̯r]), is a fresh acid-set cheese common in the Indian subcontinent (India, Bangladesh, Bhutan, Maldives, Nepal, Pakistan and Sri Lanka) made from full-fat buffalo milk or cow milk. It is a non-aged, non-melting soft cheese made by curdling milk with a fruit- or vegetable-derived acid, such as lemon juice.Paneer (pronunciado [/pəˈniːr/], ponir (pronunciáu [po̯ni̯r]), ukax mä quesito quesito acido ukampiw k'achat k'achata, India, Bangladesh, Bhutan, Maldives, Nepal, Pakistán ukat Sri Lanka) uksanakanx wali uñt'ata. Uka quesosti janiw ch'uxñatatkiti, janiw ch'uqiyatäkiti, lechex ch'uqt'atawa, jan ukax ch'uqinakat ch'uñatatatawa, leche ukamp ch'uyanchatawa.Paneer (pronunciado [/pəˈniːr /], ponir ukjamaraki uñt'ata, ukax mä quesito acido-samp k'achat ukhamawa subcontinente indiano ukanxa (India, Bangladesh, Bhutan, Maldives, Nepal, Pakistán ukat Sri Lanka) wali suma lurt'atawa. Uka quesosti janiw sinti ch'uxñatatkiti, jan ukasti lechex achunakata ukat k'allk'u manqhat luratawa.

Change 920813 had a related patch set uploaded (by Santhosh; author: Santhosh):

[mediawiki/services/machinetranslation@master] Use the correct setting for avoiding ngram repitition

https://gerrit.wikimedia.org/r/920813

Change 920813 merged by jenkins-bot:

[mediawiki/services/machinetranslation@master] Use the correct setting for avoiding ngram repitition

https://gerrit.wikimedia.org/r/920813

Change 920687 had a related patch set uploaded (by KartikMistry; author: KartikMistry):

[operations/deployment-charts@master] MinT: Update to 2023-05-18-060931-production and Set CT2_INTRA_THREADS to 0

https://gerrit.wikimedia.org/r/920687

Change 920687 merged by jenkins-bot:

[operations/deployment-charts@master] MinT: Update to 2023-05-18-060931-production and Set CT2_INTRA_THREADS to 0

https://gerrit.wikimedia.org/r/920687

Mentioned in SAL (#wikimedia-operations) [2023-05-18T11:36:45Z] <kart_> MinT: Update to 2023-05-18-060931-production and Set CT2_INTRA_THREADS to 0 (T336483)

santhosh@deploy1002:~$ curl 'https://machinetranslation.discovery.wmnet:30443/api/translate/en/ay' -X POST -H 'User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/112.0' -H 'Accept:*/*'  -H 'Accept-Encoding: gzip, deflate, br' -H 'Referer: https://translate.wmcloud.org/' -H 'Content-Type: application/json' -H 'Origin: https://translate.wmcloud.org' -H 'DNT: 1' -H 'Connection: keep-alive' -H 'Sec-Fetch-Dest: empty' -H 'Sec-Fetch-Mode: cors' -H 'Sec-Fetch-Site: same-origin' -H 'Pragma: no-cache' -H 'Cache-Control: no-cache' -H 'TE: trailers' --data-raw '{"text":"Paneer (pronounced [/pəˈniːr/]), also known as ponir (pronounced [po̯ni̯r]), is a fresh acid-set cheese common in the Indian subcontinent (India, Bangladesh, Bhutan, Maldives, Nepal, Pakistan and Sri Lanka) made from full-fat buffalo milk or cow milk. \n Itis a non-aged, non-melting soft cheese made by curdling milk with a fruit- or vegetable-derived acid, such as lemon juice."}'
`
{
"model":"nllb-wikiedia","
sourcelanguage":"en","
targetlanguage":"ay","
translation":"Paneer (pronunciado [/p\u0259\u02c8ni\u02d0r/], ponir (prononciado[po\u032fni\u032fr]), ukax m\u00e4 quesorakiwa, ukax subcontinente indiano (India, Bangladesh, Bhutan, Maldives, Nepal, Pakist\u00e1n ukat Sri Lanka) ukanx wali ch'ux\u00f1aki.\nUka quesosti janiw sinti ch'ux\u00f1atatkiti, janiw chhaqhayatakiti, lechex achunakata jan ukax ch 'ux\u00f1a quqanakat luratawa, lim\u00f3n jusi ukhama.",
"translationtime":1.1759014129638672
}

After the recent changes, the translation from MinT does no longer seem to have the previous issues of skipping parts of the translation. In fact, it seems to deal better than NLLB-200 with some aspects:

  • The parenthesis in "...Indian subcontinent (India, Bangladesh..." seems to get lost in the NLLB-200 translation, while it is preserved in MinT.
  • The repetition of words is no longer happening in MinT, while NLLB-200 still presents some instances ("quesito quesito").
Original textNLLB-200MinT
Paneer (pronounced [/pəˈniːr/]), also known as ponir (pronounced [po̯ni̯r]), is a fresh acid-set cheese common in the Indian subcontinent (India, Bangladesh, Bhutan, Maldives, Nepal, Pakistan and Sri Lanka) made from full-fat buffalo milk or cow milk. It is a non-aged, non-melting soft cheese made by curdling milk with a fruit- or vegetable-derived acid, such as lemon juice.Paneer (pronunciado [/pəˈniːr/], ponir (pronunciáu [po̯ni̯r]), ukax mä quesito quesito acido ukampiw k'achat k'achata, India, Bangladesh, Bhutan, Maldives, Nepal, Pakistán ukat Sri Lanka) uksanakanx wali uñt'ata. Uka quesosti janiw ch'uxñatatkiti, janiw ch'uqiyatäkiti, lechex ch'uqt'atawa, jan ukax ch'uqinakat ch'uñatatatawa, leche ukamp ch'uyanchatawa.Paneer (pronunciado [/pəˈniːr/], ponir (prononciado[po̯ni̯r]), ukax mä quesorakiwa, ukax subcontinente indiano (India, Bangladesh, Bhutan, Maldives, Nepal, Pakistán ukat Sri Lanka) ukanx wali ch'uxñaki. Uka quesosti janiw sinti ch'uxñatatkiti, janiw chhaqhayatakiti, lechex achunakata jan ukax ch 'uxña quqanakat luratawa, limón jusi ukhama.