| • Nikerabbit | |
| Jan 19 2022, 11:11 AM |
| F54715096: image.png | |
| Mon, Jun 10, 7:14 AM |
There is interest for having the translations available as dumps to allow efficient use of the translations in other translation memories etc. The message collection Action API interface has performance concerns that make its use not suitable for this.
We can provide dumps as files that are fast to download without using CPU resources.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | abi_ | T342427 Integrate multilingual dataset from the Translate extension into OpusMT | |||
| Resolved | Wangombe | T299493 Provide regular dumps of translations from translatewiki.net |
Change 755356 had a related patch set uploaded (by Nikerabbit; author: Nikerabbit):
[mediawiki/extensions/Translate@master] Allow exporting AggregateMessageGroups in offline format via CLI
Change 755356 merged by jenkins-bot:
[mediawiki/extensions/Translate@master] Allow exporting AggregateMessageGroups in offline format via CLI
Hello! Just for follow-up. Code was merged many months ago. Are dumps already happening? Thanks!
@Toniher I believe this got stalled on waiting feedback about which message groups to export, in combined file per language, or one file per language per group, in which format (is gettext okay?) and which frequency is sufficient for upates?
@Nikerabbit I think we already commented and Gettext would be OK, but let me ping @Txemaq so he can tell more about...
If possible, we prefer periodic dumps as follow:
Thanks in advance!
Hi @Nikerabbit ! If there is anything else you think we could help on this, please let us know! Thanks!
We're planning to automate the generation of the dumps, this would involve the following:
I used this script to generate the tarball for OpusMT team:
#!/bin/bash cd /srv/mediawiki/workdir groups=$(php maintenance/run.php ./extensions/Translate/scripts/expand-groupspec.php --exportable '*') while IFS= read -r group; do echo "Exporting $group .." php maintenance/run.php /srv/mediawiki/workdir/extensions/Translate/scripts/export.php --target /home/abi/tmp/export --group "$group" --lang "*" --offline-gettext-format "" --skip-group-sync-check done <<< "$groups"
The file can be created under /www/translatewiki.net/docroot/static/translation-dump/ with the name - translations-2024-05-09.tar.gz as an example, the info.txt file will contain the name of the latest translation dump file. Automated scripts can read the info.txt file that will always be present and will contain the name of the latest dump file.
Change #1031444 had a related patch set uploaded (by Wangombe; author: Wangombe):
[translatewiki@master] Add script to automate export of translation dumps
Change #1034837 had a related patch set uploaded (by Wangombe; author: Wangombe):
[translatewiki@master] Add automation script to export translation dumps every 6 months
Change #1031444 merged by jenkins-bot:
[translatewiki@master] Add script to automate export of translation dumps
Change #1034837 merged by jenkins-bot:
[translatewiki@master] Add automation script to export translation dumps every 6 months
Ran a test build of the script that generated an archive: https://translatewiki.net/static/translation-dump/info.txt
Deployed the script and puppet changes on translatewiki.net
Change #1038250 had a related patch set uploaded (by Nikerabbit; author: Nikerabbit):
[translatewiki@master] Add dir listing for translation dumps
Change #1038250 merged by jenkins-bot:
[translatewiki@master] Add dir listing for translation dumps
Screenshot of the extracted tarball.
the documentation is available through this link: https://translatewiki.net/wiki/Dumps
Change #1047438 had a related patch set uploaded (by Wangombe; author: Wangombe):
[translatewiki@master] Remove existing tarball if any before saving a new translation dump.
Change #1047438 merged by jenkins-bot:
[translatewiki@master] Remove existing tarball if any before saving a new translation dump.