Page MenuHomePhabricator

Improved support for transferring translations between wikis
Closed, ResolvedPublic16 Estimated Story PointsFeature

Description

Feature summary (what you would like to be able to do and where):
Right now translations can only be transferred between wikis by exporting and importing CSV files. This does not always map to the new location well, and some information is sometimes lost in the transfer. The process is also very time consuming when there are many languages translated. As we build out usage of this service on more wikis and consider doing more org communications work with MediaWiki based translation tools - it would be good to be able to move translations between wikis more easily.

Use case(s) (list the steps that you performed to discover that problem, and describe the actual underlying problem which you want to solve. Do not describe only a solution):

Benefits (why should this be implemented?):

Time saved. (Possibly) attribution and other metadata preserved.

Event Timeline

Nikerabbit raised the priority of this task from Low to High.Apr 4 2023, 12:02 PM
Nikerabbit set the point value for this task to 16.Apr 4 2023, 12:15 PM

We discussed this task with @Varnent for further understanding of their use case. Here are some notes from that discussion:

  1. Initially a solution that speeds up the process of transferring translations is wanted, and then step by step improvements can be made.
  2. In the long term, it would be good to have attribution and the entire revision history for the translations

For this purpose, a new file type if required can be used but it should be easy to open and examine. Since exporting / importing a file containing all revisions, attributions for a message group would be a time consuming process, the solution might involve creating a maintenance script or a job that runs in the background.

Anyone should be able to perform exports, but imports should be limited to translation administrators.


Considering the requirements and the need to be able to export and import attributions in the future, we're been looking at WikiExporter and WikiImporter classes that are used on Special:Export & Special:Import pages.

Change 907905 had a related patch set uploaded (by Abijeet Patro; author: Abijeet Patro):

[mediawiki/extensions/Translate@master] Add script to export an entiry translatable bundle

https://gerrit.wikimedia.org/r/907905

Change 916413 had a related patch set uploaded (by Abijeet Patro; author: Abijeet Patro):

[mediawiki/extensions/Translate@master] Add script to export an entire translatable bundle

https://gerrit.wikimedia.org/r/916413

FYI that @Xeno_WMF is working on a migration of translations right now in case you want another perspective on the process.

Change 923753 had a related patch set uploaded (by Abijeet Patro; author: Abijeet Patro):

[mediawiki/extensions/Translate@master] Allow export/import of translatable bundle from STDOUT and STDIN

https://gerrit.wikimedia.org/r/923753

Change 907905 merged by jenkins-bot:

[mediawiki/extensions/Translate@master] Add script to export an entire translatable bundle

https://gerrit.wikimedia.org/r/907905

Change 916413 merged by jenkins-bot:

[mediawiki/extensions/Translate@master] Add script to import an entire translatable bundle

https://gerrit.wikimedia.org/r/916413

I deployed and tested this on translatewiki. I exported a page from translatewiki and imported it on my local machine and also did the reverse.

Export from translatewiki and import on my local wiki worked fine. But I encountered an unexpected behavior when exporting from my local machine and importing translatewiki. Import Log on translatewiki: https://translatewiki.net/w/i.php?title=Special:Log&logid=8122320

The page I exported from my local wiki was in the NS_MAIN namespace but when importing on translatewiki, I wanted to do it under my user page (User:Abijeet_Patro), so I used that as the root-page when importing.
The import process on translatewiki.net completed successfully, and the translatable page, and its translations were created as expected but the translation units were created in an incorrect location.

Expected import location: https://translatewiki.net/w/i.php?title=Translations:User:Abijeet_Patro/Translations:TwnImportTest/1/en
Actual import location: https://translatewiki.net/wiki/User:Abijeet_Patro/Translations:TwnImportTest/1/en

In summary, I think exporting and importing under the same namespace work as expected.

@Varnent We've added two scripts to export and import translatable bundles from one wiki to another. You can read more about it here: https://www.mediawiki.org/wiki/User:APatro_(WMF)/Move_translatable_bundles_across_wiki

Please note the current limitation of the first version of this script:

  • This only tags the translation, but does not mark the page for translation. That has to be done manually
  • History / Revisions from the source wiki are not imported to the target wiki.
  • For translatable pages, metadata such as priority languages are not exported / imported. Please add them when marking the page for translation.

We've built this on top of the MediaWiki import / export XML format so we should be able to add attributions, history, revisions in the future.

Nikerabbit changed the task status from Open to In Progress.Jul 4 2023, 8:40 AM

@Varnent We've added two scripts to export and import translatable bundles from one wiki to another. You can read more about it here: https://www.mediawiki.org/wiki/User:APatro_(WMF)/Move_translatable_bundles_across_wiki

Please note the current limitation of the first version of this script:

  • This only tags the translation, but does not mark the page for translation. That has to be done manually
  • History / Revisions from the source wiki are not imported to the target wiki.
  • For translatable pages, metadata such as priority languages are not exported / imported. Please add them when marking the page for translation.

We've built this on top of the MediaWiki import / export XML format so we should be able to add attributions, history, revisions in the future.

I've identified follow ups,

  1. T347153: Add option to set the target namespace to import translatable bundles to
  2. T347154: Add option to set the target pagename when importing translatable bundles
  3. T347155: Import metadata and automatically mark page for translation when importing translatable bundles
  4. T347156: Transfer history and revision information when moving translatable bundles between wikis

We can triage these tasks after talking to the comms team and understanding their needs.

I'm marking this task as done for now.