Page MenuHomePhabricator

Support export for offline translation also from command line
Closed, ResolvedPublic

Description

The Specials page for exporting mediawiki translation data for "off-line" translation can and does export data contained in the WikiPageMessageGroup type, however the command-line tool for this same functionality will produce an error "EE2: Unexportable message group page-SomePage"

The script at scripts/export.php seems to only work with "FileBasedMessageGroup" types and there is no clear documentation of this or how to convert between the two.

Is there a quick patch I can make to this export tool that would allow me to create the .po files with it the same way the Specials page does?

Documentation to update:

Related Objects

Event Timeline

TheFae created this task.Mar 20 2020, 8:37 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 20 2020, 8:37 PM

You would do something like this:

diff --git a/messagegroups/FileBasedMessageGroup.php b/messagegroups/FileBasedMessageGroup.php
index 0ac77f21d..16173f9d5 100644
--- a/messagegroups/FileBasedMessageGroup.php
+++ b/messagegroups/FileBasedMessageGroup.php
@@ -35,7 +35,7 @@ class FileBasedMessageGroup extends MessageGroupBase implements MetaYamlSchemaEx
                        ],
                        'FILES' => [
                                'sourcePattern' => '',
-                               'targetPattern' => '',
+                               'targetPattern' => "{$group->getId()}/%CODE%.po",
                        ],
                ];
 
diff --git a/scripts/export.php b/scripts/export.php
index 4b0724640..02edfd92c 100644
--- a/scripts/export.php
+++ b/scripts/export.php
@@ -93,6 +93,12 @@ class CommandlineExport extends Maintenance {
                        false, /*required*/
                        false /*has arg*/
                );
+               $this->addOption(
+                       'offline-gettext-format',
+                       '(optional) Export languages in offline Gettext format',
+                       false, /*required*/
+                       false /*has arg*/
+               );
                $this->requireExtension( 'Translate' );
        }
 
@@ -127,6 +133,7 @@ class CommandlineExport extends Maintenance {
                $reqLangs = array_flip( $reqLangs );
 
                $codemapOnly = $this->hasOption( 'codemaponly' );
+               $forOffline = $this->hasOption('offline-gettext-format' );
 
                $groupIds = explode( ',', trim( $this->getOption( 'group' ) ) );
                $groupIds = MessageGroups::expandWildcards( $groupIds );
@@ -141,7 +148,7 @@ class CommandlineExport extends Maintenance {
                                continue;
                        }
 
-                       if ( !$group instanceof FileBasedMessageGroup ) {
+                       if ( !$forOffline && !$group instanceof FileBasedMessageGroup ) {
                                $this->output( "EE2: Unexportable message group $groupId.\n" );
                                unset( $groups[$groupId] );
                                continue;
@@ -257,7 +264,14 @@ class CommandlineExport extends Maintenance {
 
                        $this->output( 'Exporting ' . count( $langs ) . " languages for group $groupId" );
 
-                       $ffs = $group->getFFS();
+                       if ( $forOffline ) {
+                               $fileBasedGroup = FileBasedMessageGroup::newFromMessageGroup( $group );
+                               $ffs = new GettextFFS( $fileBasedGroup );
+                               $ffs->setOfflineMode( true );
+                       } else {
+                               $ffs = $group->getFFS();
+                       }
+
                        $ffs->setWritePath( $target );
                        $sourceLanguage = $group->getSourceLanguage();
                        $collection = $group->initCollection( $sourceLanguage );

Thanks for the quick response!
Can we expect this to be added to a future version of Translate?

I ask because we plan to use it for automation of exports and I want to make sure we can easily update the extension without needing to patch this in every time we update. Updating MW is already an involved sort of thing, but I may not always be the only one doing this sort of work for the project.

Thanks again for your help with this!

Nikerabbit renamed this task from scripts/export.php does not export due to WikiPageMessageGroup to Support export for offline translation also from command line.Mar 23 2020, 10:38 AM

Change 582781 had a related patch set uploaded (by Nikerabbit; owner: Nikerabbit):
[mediawiki/extensions/Translate@master] export.php: add --offline-gettext-format argument

https://gerrit.wikimedia.org/r/582781

Nikerabbit triaged this task as Low priority.Mar 23 2020, 10:45 AM

I pushed the patch to code review.

I would also like to request adding a command line argument or some method to set the targetPattern element of the protected $conf variable, if possible.

Adding a %GROUPID% replacement variable and some method to set the target file pattern would enable creating the .po files with the same name format as those produced by the Specials export page, and other customization of the output file names.

Sorry to move the goal post a tad here. Thanks again for the support and quick patching!

Change 582781 merged by jenkins-bot:
[mediawiki/extensions/Translate@master] export.php: add --offline-gettext-format argument

https://gerrit.wikimedia.org/r/582781

I'll see if it is easy to implement targetPattern override.

TheFae added a comment.Apr 2 2020, 3:26 PM

I've managed to add such a feature. I was trying to add it to the proposed changes but couldn't figure out how to do so. This week has been sad and exhausting so I've shelved my volunteer efforts until i'm feeling able again.

Thanks for your help with all this.
Be safe, and let your loved ones know you love them.

Change 585678 had a related patch set uploaded (by Nikerabbit; owner: Nikerabbit):
[mediawiki/extensions/Translate@master] export.php: Make FILES.targetPattern configurable

https://gerrit.wikimedia.org/r/585678

Change 585678 merged by jenkins-bot:
[mediawiki/extensions/Translate@master] export.php: Make FILES.targetPattern configurable

https://gerrit.wikimedia.org/r/585678

abi_ added a subscriber: abi_.

Related patches have been merged, can wait for a sanity check to ensure exports are working well. We can resolve this after twn deployment and export run on Thursday.