Page MenuHomePhabricator

Bug with importing fuzzy translations from .po files
Closed, ResolvedPublicBUG REPORT

Description

Steps to replicate the issue (include links if applicable):

  • Export translations contains any fuzzy translations into gettext .po file with Special:ExportTranslations page, then import it back with Special:ImportTranslations page without any changes.

What happens?:

There is a significant bug in the extension which makes it impossible to fully use the Special:ImportTranslations function with the gettext format in cases where the imported po-file contains strings with fuzzy translations. When lines containing, e.g.:

#, fuzzy
msgctxt "Help:Requests_for_deletions/1"
msgid "Items which do not meet [[$1|Wikidata's notability policy]] can be deleted. Please nominate items for deletions on [[Wikidata:Requests for deletions]] under the \"Requests\" section below. If it is obvious vandalism, just add the page directly (''[[$2|gadget]] available''), or ping an [[$3|administrator]] to delete it. Contact can also be made with an administrator in $4."
msgstr "Элементы, не отвечающие [[$1|правилам Викиданных о значимости]], могут быть удалены. Пожалуйста, оставляйте здесь заявки на удаление элементов под заголовком «Запросы» ниже. Если это очевидный вандализм, просто добавьте страницу сюда (''доступен [[$2|гаджет]]'') либо попросите [[$3|администратора]] удалить её. Можно также связаться с администратором с помощью $4."

are imported, instead of being marked as "Import and fuzzy", as they should be, the label "!!FUZZY!!" is added directly before the translation text. This causes already existing translations marked as fuzzy to duplicate this tag, which turns into "!!FUZZY!!!!FUZZY!!" in the text and ruins the translation. As an example, you can see it here: https://www.wikidata.org/w/index.php?title=Translations:Help:Requests_for_deletions/1/ru&type=revision&diff=1774680393&oldid=1299499243&diffmode=source

Please fix this behavior in the extension and make sure that fuzzy translations are marked with Special:ImportTranslations page interface instead of adding prefix label.

What should have happened instead?:

Translations should be imported without additional "!!FUZZY!!" mark if it is already exist in text.

Event Timeline

Evs updated the task description. (Show Details)

As i see in the scripts/poimport.php, the importing of fuzzy translations strings is doing with such code:

			// Fuzzy messages
			if ( preg_match( '/^#, fuzzy$/m', $section ) ) {
				$translation = TRANSLATE_FUZZY . $translation;
			}

This is categorically wrong, because this kind of "dirty hack" does not recognize the fact that existing translations can be already marked as fuzzy, which causes duplicate !!FUZZY!! tags. Marking should not be done by the import script in this way, but with src/Synchronization/MessageWebImporter.php, where it should be done with the translate-manage-action-conflict as default action for fuzzy strings.

Wangombe changed the task status from Open to In Progress.Jan 9 2023, 8:42 AM
Wangombe claimed this task.
Nikerabbit triaged this task as Medium priority.Jan 12 2023, 2:38 PM
Wangombe changed the task status from In Progress to Open.Jan 19 2023, 7:43 AM

So far, while the issue can be reproduced, we haven't been able to identify where the second !!FUZZY!! is added from. The poimport.php file is not called as the user interacts with the Special:ImportTranslations page so that might not be the culprit we are looking for. Will keep investigating.

So far, while the issue can be reproduced, we haven't been able to identify where the second !!FUZZY!! is added from. The poimport.php file is not called as the user interacts with the Special:ImportTranslations page so that might not be the culprit we are looking for. Will keep investigating.

But what is the purpose of poimport.php file? :)

The poimport.php file is used with the command line and not with the Special:ImportTranslations page.

Change 896012 had a related patch set uploaded (by Abijeet Patro; author: Abijeet Patro):

[mediawiki/extensions/Translate@master] ImportTranslations: Do not add FUZZY if message already has fuzzy tag

https://gerrit.wikimedia.org/r/896012

Change 896092 had a related patch set uploaded (by Abijeet Patro; author: Abijeet Patro):

[mediawiki/extensions/Translate@master] LegacyInterfaceHookHandler: Check before adding FUZZY string

https://gerrit.wikimedia.org/r/896092

Change 896093 had a related patch set uploaded (by Abijeet Patro; author: Abijeet Patro):

[mediawiki/extensions/Translate@master] MessageWebImporter: Add fuzzy in diff for old messages with fuzzy tag

https://gerrit.wikimedia.org/r/896093

Change 896012 merged by jenkins-bot:

[mediawiki/extensions/Translate@master] ImportTranslations: Do not add FUZZY if message already has fuzzy tag

https://gerrit.wikimedia.org/r/896012

Change 896092 merged by jenkins-bot:

[mediawiki/extensions/Translate@master] LegacyInterfaceHookHandler: Check before adding FUZZY string

https://gerrit.wikimedia.org/r/896092

Change 896093 merged by jenkins-bot:

[mediawiki/extensions/Translate@master] MessageWebImporter: Add fuzzy in diff for old messages with fuzzy tag

https://gerrit.wikimedia.org/r/896093

Change 902403 had a related patch set uploaded (by Abijeet Patro; author: Abijeet Patro):

[mediawiki/extensions/Translate@master] MessageWebImporter: Use translation instead of language code on import

https://gerrit.wikimedia.org/r/902403

Change 902370 had a related patch set uploaded (by Abijeet Patro; author: Abijeet Patro):

[mediawiki/extensions/Translate@wmf/1.41.0-wmf.1] MessageWebImporter: Use translation instead of language code on import

https://gerrit.wikimedia.org/r/902370

Change 902403 merged by jenkins-bot:

[mediawiki/extensions/Translate@master] MessageWebImporter: Use translation instead of language code on import

https://gerrit.wikimedia.org/r/902403

Change 902370 merged by jenkins-bot:

[mediawiki/extensions/Translate@wmf/1.41.0-wmf.1] MessageWebImporter: Use translation instead of language code on import

https://gerrit.wikimedia.org/r/902370

Mentioned in SAL (#wikimedia-operations) [2023-03-23T20:22:59Z] <taavi@deploy2002> Started scap: Backport for [[gerrit:902370|MessageWebImporter: Use translation instead of language code on import (T323430)]]

Mentioned in SAL (#wikimedia-operations) [2023-03-23T20:24:35Z] <taavi@deploy2002> abi and taavi: Backport for [[gerrit:902370|MessageWebImporter: Use translation instead of language code on import (T323430)]] synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet

Mentioned in SAL (#wikimedia-operations) [2023-03-23T20:33:56Z] <taavi@deploy2002> Finished scap: Backport for [[gerrit:902370|MessageWebImporter: Use translation instead of language code on import (T323430)]] (duration: 10m 56s)

The Movement strategy team found an issue while testing this feature that has since been fixed. No issues found on retest.