Page MenuHomePhabricator

MinT leaks template metadata in translated content
Closed, ResolvedPublic

Assigned To
Authored By
Pginer-WMF
Jun 26 2024, 10:20 AM
Referenced Files
F57617053: 2024-10-15_08-15-31.png
Oct 15 2024, 3:23 PM
F57617049: 2024-10-15_08-12-44.png
Oct 15 2024, 3:23 PM
F57617051: 2024-10-15_08-12-04.png
Oct 15 2024, 3:23 PM
F57600286: image.png
Oct 9 2024, 7:21 AM
F57600284: image.png
Oct 9 2024, 7:21 AM
F57524710: 2024-09-19_15-26-05.png
Sep 19 2024, 10:49 PM
F57524707: 2024-09-19_15-13-36.png
Sep 19 2024, 10:49 PM
F57524705: 2024-09-19_15-13-45.png
Sep 19 2024, 10:49 PM

Description

The initial text paragraph of the Tardigrade article presents issues when translated using MinT. This can be checked in different MinT-based products:

In both cases, the translation from MinT seems to be filled with metadata coming from a template. In the case of Content Translation, the issue is not happening when using Google Translate as a translation service, as illustrated below. The wikitext for the problematic paragraph has been captured in this test page and you can start the translation for the problematic content.

Derived Requirement

Ensure that the initial paragraph of the Tardigrade article is correctly translated by MinT without including metadata from a template in both Content Translation and MinT for Wiki Readers.

BDD

Feature: Accurate translation of the Tardigrade article using MinT

Scenario: Ensure the initial paragraph of the Tardigrade article is correctly translated using MinT

Given the user is translating the Tardigrade article from English to Igbo using Content Translation or MinT for Wiki Readers
When the user views the translated content
Then the initial paragraph of the Tardigrade article should not include metadata from a template
Test Steps

Test Case 1: Ensure the initial paragraph of the Tardigrade article is correctly translated in Content Translation

Open Wikipedia and navigate to the Content Translation page for the Tardigrade article.
Select MinT as the translation service.
Check the translation of the initial paragraph of the Tardigrade article.
✅❓❌⬜ AC1: Confirm that the initial paragraph does not include metadata from a template in the translated content.

Test Case 2: Ensure the initial paragraph of the Tardigrade article is correctly translated in MinT for Wiki Readers

Open the MinT for Wiki Readers link for the Tardigrade article.
Check the translation of the initial paragraph of the Tardigrade article.
✅❓❌⬜ AC2: Confirm that the initial paragraph does not include metadata from a template in the translated content.

QA Results - Special:ContentTranslation & Automatic Translation

ACStatusDetails
1T368521#10163003
1T368521#10163003

Event Timeline

Pginer-WMF triaged this task as Medium priority.
Pginer-WMF added a project: MinT.
Pginer-WMF added subscribers: ngkountas, santhosh.

Change #1050950 had a related patch set uploaded (by Santhosh; author: Santhosh):

[mediawiki/services/machinetranslation@master] html: Contrain the fuzzy matching to avoid broken html

https://gerrit.wikimedia.org/r/1050950

Change #1050950 merged by jenkins-bot:

[mediawiki/services/machinetranslation@master] html: Constrain the fuzzy matching to avoid broken html

https://gerrit.wikimedia.org/r/1050950

Change #1057421 had a related patch set uploaded (by KartikMistry; author: KartikMistry):

[operations/deployment-charts@master] Update MinT to 2024-07-24-145137-production

https://gerrit.wikimedia.org/r/1057421

Change #1057421 merged by jenkins-bot:

[operations/deployment-charts@master] Update MinT to 2024-08-05-062247-production

https://gerrit.wikimedia.org/r/1057421

@santhosh Please review the issues below, thanks!

Status: ❌ FAIL
Environment: Special:ContentTranslation &Automatic Translation
OS: macOS Sonoma 14.6.1
Browser: Chrome 128
Device: MBA
Emulated Device: NA

Test Artifact(s):

https://test.wikipedia.org/w/index.php?title=Special:ContentTranslation&from=en&to=ig&page=Firefox
https://ig.wikipedia.org/wiki/Special:ContentTranslation?from=en&to=ig&page=Tardigrade&targettitle=Tardigrade&campaign=undefined
https://ig.m.wikipedia.org/wiki/Ih%C3%BC_k%C3%A1r%C3%ADr%C3%AD:AutomaticTranslation?page=Tardigrade&from=en&to=ig&step=translation

Test Steps

Test Case 1: Ensure the initial paragraph of the Tardigrade article is correctly translated in Content Translation

Open Wikipedia and navigate to the Content Translation page for the Tardigrade article.
Select MinT as the translation service.
Check the translation of the initial paragraph of the Tardigrade article.
❌AC1: Confirm that the initial paragraph does not include metadata from a template in the translated content.

https://ig.wikipedia.org/wiki/Special:ContentTranslation?from=en&to=ig&page=Tardigrade&targettitle=Tardigrade&campaign=undefined

❌Mint✅Google
2024-09-19_15-13-45.png (1,628×1,124 px, 320 KB)
2024-09-19_15-13-36.png (1,617×1,110 px, 326 KB)

Test Case 2: Ensure the initial paragraph of the Tardigrade article is correctly translated in MinT for Wiki Readers

  1. Open the MinT for Wiki Readers link for the Tardigrade article.
  2. Check the translation of the initial paragraph of the Tardigrade article.
  3. ❌ AC2: Confirm that the initial paragraph does not include metadata from a template in the translated content.

https://ig.m.wikipedia.org/wiki/Ih%C3%BC_k%C3%A1r%C3%ADr%C3%AD:AutomaticTranslation?page=Tardigrade&from=en&to=ig&step=translation

Mint
2024-09-19_15-26-05.png (1,293×515 px, 152 KB)

Trying to debug the issue. It seems the issue is happening with nllb-wikipedia(default) model and not with nllb-600m model.

image.png (1,224×467 px, 149 KB)
image.png (1,224×467 px, 112 KB)

Checking what is the difference.

Change #1078930 had a related patch set uploaded (by Santhosh; author: Santhosh):

[mediawiki/services/machinetranslation@master] html: Fix issue with fuzzy matching with non-zero search_start

https://gerrit.wikimedia.org/r/1078930

Change #1078930 merged by jenkins-bot:

[mediawiki/services/machinetranslation@master] html: Fix issue with fuzzy matching with non-zero search_start

https://gerrit.wikimedia.org/r/1078930

Change #1079682 had a related patch set uploaded (by KartikMistry; author: KartikMistry):

[operations/deployment-charts@master] Update MinT to 2024-10-11-113932-production

https://gerrit.wikimedia.org/r/1079682

Change #1079682 merged by jenkins-bot:

[operations/deployment-charts@master] Update MinT to 2024-10-11-113932-production

https://gerrit.wikimedia.org/r/1079682

@santhosh Mint seems to be fixed now but check out the screenshots from Current Google and Previous Google. The Previous Google screenshot did not have <link> as seen in the Current Google screenshot below.

Status: ❌ AC1- Current Google
Environment: Special:ContentTranslation &Automatic Translation
OS: macOS Sonoma 14.6.1
Browser: Chrome 128
Device: MBA
Emulated Device: NA

Test Artifact(s):

https://test.wikipedia.org/w/index.php?title=Special:ContentTranslation&from=en&to=ig&page=Firefox
https://ig.wikipedia.org/wiki/Special:ContentTranslation?from=en&to=ig&page=Tardigrade&targettitle=Tardigrade&campaign=undefined
https://ig.m.wikipedia.org/wiki/Ih%C3%BC_k%C3%A1r%C3%ADr%C3%AD:AutomaticTranslation?page=Tardigrade&from=en&to=ig&step=translation

Test Steps

Test Case 1: Ensure the initial paragraph of the Tardigrade article is correctly translated in Content Translation

Open Wikipedia and navigate to the Content Translation page for the Tardigrade article.
Select MinT as the translation service.
Check the translation of the initial paragraph of the Tardigrade article.
❌ AC1: Confirm that the initial paragraph does not include metadata from a template in the translated content.

https://ig.wikipedia.org/wiki/Special:ContentTranslation?from=en&to=ig&page=Tardigrade&targettitle=Tardigrade&campaign=undefined

✅ Mint✅ Previous Google❌ Current Google
2024-10-15_08-12-04.png (1,696×1,003 px, 359 KB)
2024-09-19_15-13-36.png (1,617×1,110 px, 326 KB)
2024-10-15_08-12-44.png (1,702×1,009 px, 382 KB)

Test Case 2: Ensure the initial paragraph of the Tardigrade article is correctly translated in MinT for Wiki Readers

  1. Open the MinT for Wiki Readers link for the Tardigrade article.
  2. Check the translation of the initial paragraph of the Tardigrade article.
  3. ✅ AC2: Confirm that the initial paragraph does not include metadata from a template in the translated content.

https://ig.m.wikipedia.org/wiki/Ih%C3%BC_k%C3%A1r%C3%ADr%C3%AD:AutomaticTranslation?page=Tardigrade&from=en&to=ig&step=translation

Mint
2024-10-15_08-15-31.png (1,316×433 px, 97 KB)
GMikesell-WMF changed the task status from Open to In Progress.Oct 15 2024, 3:46 PM

The Google translation issue seems a valid, but it is totally unrelated to the original one. Google translation and markup fixes are handles by CX-cxserver project and MinT has no code related to Google.

I create a new ticket to track that T377305: Corrupted HTML with Google translation in en to ig translation. As this specific issue has been verified as fixed, moving to done