Page MenuHomePhabricator

Support doing translation update backports to stable branches
Closed, ResolvedPublicFeature

Description

Long time ago we used to do translation backports so that MediaWiki point releases would get additional translations. We had a system where we actually supported translating messages only present in those older versions but not in the latest version, but it was discontinued. It required quite a bit of manual effort to keep it up to date. While Wikimedia sites have weekly deployment cycle, third party wikis do not get updated translations unless the upgrade to a new major release. They can run LocalisationUpdate, but in practice nobody does because it needs manual setup and is very slow (if used against github) or takes a lot of disk space if using with local clones of everything).

By doing LocalisationUpdate style backports to the stable branches regularly, third party users who update to MediaWiki point releases or run from the stable branch would get more frequent translation updates without any additional work from their side.

If implemented in the Translate extension, we can make this support generic enough to work with different version control systems and file formats.

The backport logic is rather simple. We will have two versions of files checked out: the main branch and the stable branch. To backport to multiple stable branches, we just repeat the following process for each:

  1. Parse definitions in the main branch
  2. Parse definitions in the stable branch
  3. Annotate the list of keys in the stable branch whether they are compatible (can use === comparison for simplicity and safety)
  4. For each language:
    1. Parse the translation in the stable branch
    2. Iterate over the annotated list of keys:
      1. If keys are compatible, take first available translation in the order of main branch, stable branch, none.
      2. If keys are incompatible, take translation from the stable branch, if available.
    3. Generate translation file from this list of translations

The main user interface for this would be a new command autobackport(-mediawiki) that works similarly to autoexport(-mediawiki).

When repong instructed to do backports, instead of updating the main branch, it would iterate over the backport branches. For update, it would clone or update the repo on REPONG-ROOT/backports/mediawiki/REL1_34 (repeat with all configured branches). State synchronization is not available since source and target branches are different. diff/status/commit commands would work as usual, just iterating over these branches. export command is tricker. I propose to create a new script in Translate called backport.php that would be used like this: php backport.php --target=REPONG-ROOT --targetPath=mediawiki --targetBranch=REL1_34 and the usual parameters about groups, thresholds, languages etc. Since we do not have mapping from a repo to message group (only form project to message group), we have to use the targetPath parameter to find out which message groups we should do. For these message groups, we construct the FFS objects for parsing and executing the algorithm detailed earlier. There is no need to load messages from the database.

Event Timeline

Change 692604 had a related patch set uploaded (by Nikerabbit; author: Nikerabbit):

[mediawiki/extensions/Translate@master] Add backport.php

https://gerrit.wikimedia.org/r/692604

Change 692604 merged by jenkins-bot:

[mediawiki/extensions/Translate@master] Add backport.php

https://gerrit.wikimedia.org/r/692604

Change 699745 had a related patch set uploaded (by Nikerabbit; author: Nikerabbit):

[translatewiki@master] repong: Add backport command

https://gerrit.wikimedia.org/r/699745

Change 706489 had a related patch set uploaded (by Nikerabbit; author: Nikerabbit):

[mediawiki/extensions/Translate@master] BackportTranslationsMaintenanceScript: Add more filters and improve output

https://gerrit.wikimedia.org/r/706489

Nikerabbit changed the subtype of this task from "Task" to "Feature Request".Jul 28 2021, 7:49 AM

Change 706489 merged by jenkins-bot:

[mediawiki/extensions/Translate@master] BackportTranslationsMaintenanceScript: Add more filters and improve output

https://gerrit.wikimedia.org/r/706489

Change 699745 merged by jenkins-bot:

[translatewiki@master] repong: Add backport command

https://gerrit.wikimedia.org/r/699745

Change 716214 had a related patch set uploaded (by Nikerabbit; author: Nikerabbit):

[translatewiki@master] repomulti: add support for backport command

https://gerrit.wikimedia.org/r/716214

Change 720949 had a related patch set uploaded (by Nikerabbit; author: Nikerabbit):

[translatewiki@master] Commit support for backports

https://gerrit.wikimedia.org/r/720949

Change 720959 had a related patch set uploaded (by Nikerabbit; author: Nikerabbit):

[mediawiki/extensions/Translate@master] Avoid duplicated directory separators for premade groups

https://gerrit.wikimedia.org/r/720959

Change 720968 had a related patch set uploaded (by Nikerabbit; author: Nikerabbit):

[mediawiki/extensions/Translate@master] Backports: Preserve order of message keys

https://gerrit.wikimedia.org/r/720968

Change 720959 merged by jenkins-bot:

[mediawiki/extensions/Translate@master] Avoid duplicated directory separators for premade groups

https://gerrit.wikimedia.org/r/720959

Change 716214 merged by jenkins-bot:

[translatewiki@master] repomulti: add support for backport command

https://gerrit.wikimedia.org/r/716214

Change 720968 merged by jenkins-bot:

[mediawiki/extensions/Translate@master] Backports: Preserve order of message keys

https://gerrit.wikimedia.org/r/720968

Nikerabbit changed the task status from Open to In Progress.Sep 25 2021, 8:56 AM

Change 720949 merged by jenkins-bot:

[translatewiki@master] Commit support for backports

https://gerrit.wikimedia.org/r/720949

Change 727292 had a related patch set uploaded (by Nikerabbit; author: Nikerabbit):

[translatewiki@master] repong: improve usability

https://gerrit.wikimedia.org/r/727292

Change 727292 merged by jenkins-bot:

[translatewiki@master] repong: improve usability

https://gerrit.wikimedia.org/r/727292

Change 734256 had a related patch set uploaded (by Nikerabbit; author: Nikerabbit):

[translatewiki@master] Support committing backports

https://gerrit.wikimedia.org/r/734256

Change 734256 merged by jenkins-bot:

[translatewiki@master] Support committing backports

https://gerrit.wikimedia.org/r/734256

Change 739224 had a related patch set uploaded (by Nikerabbit; author: Nikerabbit):

[mediawiki/extensions/Translate@master] BackportTranslationsMaintenanceScript: correctly process $group->getTranslatableLanguages

https://gerrit.wikimedia.org/r/739224

Change 739231 had a related patch set uploaded (by Nikerabbit; author: Nikerabbit):

[translatewiki@master] clupdate-git-repo: prune branches while fetching

https://gerrit.wikimedia.org/r/739231

Change 739232 had a related patch set uploaded (by Nikerabbit; author: Nikerabbit):

[mediawiki/extensions/Translate@master] BackportTranslationsMaintenanceScript: reduce noise in output

https://gerrit.wikimedia.org/r/739232

I did try to run a full backport today. Skins and core look fine. Extensions discovered two problems for which I have submitted fixes.

Change 739236 had a related patch set uploaded (by Nikerabbit; author: Nikerabbit):

[translatewiki@master] autobackport-mediawiki: make executable

https://gerrit.wikimedia.org/r/739236

Change 739236 merged by jenkins-bot:

[translatewiki@master] autobackport-mediawiki: make executable

https://gerrit.wikimedia.org/r/739236

Change 739224 merged by jenkins-bot:

[mediawiki/extensions/Translate@master] BackportTranslationsMaintenanceScript: correctly process $group->getTranslatableLanguages

https://gerrit.wikimedia.org/r/739224

Change 739232 merged by jenkins-bot:

[mediawiki/extensions/Translate@master] BackportTranslationsMaintenanceScript: reduce noise in output

https://gerrit.wikimedia.org/r/739232

Change 739231 merged by jenkins-bot:

[translatewiki@master] clupdate-git-repo: prune branches while fetching

https://gerrit.wikimedia.org/r/739231

Change 740627 had a related patch set uploaded (by Nikerabbit; author: Nikerabbit):

[mediawiki/extensions/Translate@master] JsonFFS: Fix unmangling of message keys during backports

https://gerrit.wikimedia.org/r/740627

Change 740627 merged by jenkins-bot:

[mediawiki/extensions/Translate@master] JsonFFS: Fix unmangling of message keys during backports

https://gerrit.wikimedia.org/r/740627

Did a successful run of autoexport-mediawiki REL1_37. A pending step is scheduling automatic runs for supported branches, but I'll do that as part of another task.