Page MenuHomePhabricator

Support doing translation update backports to stable branches
Open, In Progress, LowPublicFeature

Description

Long time ago we used to do translation backports so that MediaWiki point releases would get additional translations. We had a system where we actually supported translating messages only present in those older versions but not in the latest version, but it was discontinued. It required quite a bit of manual effort to keep it up to date. While Wikimedia sites have weekly deployment cycle, third party wikis do not get updated translations unless the upgrade to a new major release. They can run LocalisationUpdate, but in practice nobody does because it needs manual setup and is very slow (if used against github) or takes a lot of disk space if using with local clones of everything).

By doing LocalisationUpdate style backports to the stable branches regularly, third party users who update to MediaWiki point releases or run from the stable branch would get more frequent translation updates without any additional work from their side.

If implemented in the Translate extension, we can make this support generic enough to work with different version control systems and file formats.

The backport logic is rather simple. We will have two versions of files checked out: the main branch and the stable branch. To backport to multiple stable branches, we just repeat the following process for each:

  1. Parse definitions in the main branch
  2. Parse definitions in the stable branch
  3. Annotate the list of keys in the stable branch whether they are compatible (can use === comparison for simplicity and safety)
  4. For each language:
    1. Parse the translation in the stable branch
    2. Iterate over the annotated list of keys:
      1. If keys are compatible, take first available translation in the order of main branch, stable branch, none.
      2. If keys are incompatible, take translation from the stable branch, if available.
    3. Generate translation file from this list of translations

The main user interface for this would be a new command autobackport(-mediawiki) that works similarly to autoexport(-mediawiki). This script would call repong.php commands with a new command line parameter --backports or by duplicating the set of commands like update-backports, export-backports, etc. depending on which is a cleaner implementation.

repoconfig.yaml would be extended to support a new per-repo configuration backport-branches, for example:

mediawiki:
  group: core,ext-installer,mediawiki-api,mediawiki-rest-api,mediawiki-exif,wikimedia-paramvalidator
  auto-merge: mediawiki/core
  repos:
    mediawiki:
      type: wmgerrit
      url: https://gerrit.wikimedia.org/r/mediawiki/core
      url|export: ssh://l10n-bot@gerrit.wikimedia.org:29418/mediawiki/core
      backport-branches: [ REL1_34, REL1_35 ]

When repong instructed to do backports, instead of updating the main branch, it would iterate over the backport branches. For update, it would clone or update the repo on REPONG-ROOT/backports/mediawiki/REL1_34 (repeat with all configured branches). State synchronization is not available since source and target branches are different. diff/status/commit commands would work as usual, just iterating over these branches. export command is tricker. I propose to create a new script in Translate called backport.php that would be used like this: php backport.php --target=REPONG-ROOT --targetPath=mediawiki --targetBranch=REL1_34 and the usual parameters about groups, thresholds, languages etc. Since we do not have mapping from a repo to message group (only form project to message group), we have to use the targetPath parameter to find out which message groups we should do. For these message groups, we construct the FFS objects for parsing and executing the algorithm detailed earlier. There is no need to load messages from the database.

If more flexibility is needed, repong could take a --only-branch command that would restrict which branch to do the backports in. This would be helpful for doing on-demand backports before a release, for example.

Event Timeline

Change 692604 had a related patch set uploaded (by Nikerabbit; author: Nikerabbit):

[mediawiki/extensions/Translate@master] Add backport.php

https://gerrit.wikimedia.org/r/692604

Change 692604 merged by jenkins-bot:

[mediawiki/extensions/Translate@master] Add backport.php

https://gerrit.wikimedia.org/r/692604

Change 699745 had a related patch set uploaded (by Nikerabbit; author: Nikerabbit):

[translatewiki@master] repong: Add backport command

https://gerrit.wikimedia.org/r/699745

Change 706489 had a related patch set uploaded (by Nikerabbit; author: Nikerabbit):

[mediawiki/extensions/Translate@master] BackportTranslationsMaintenanceScript: Add more filters and improve output

https://gerrit.wikimedia.org/r/706489

Nikerabbit changed the subtype of this task from "Task" to "Feature Request".Jul 28 2021, 7:49 AM

Change 706489 merged by jenkins-bot:

[mediawiki/extensions/Translate@master] BackportTranslationsMaintenanceScript: Add more filters and improve output

https://gerrit.wikimedia.org/r/706489

Change 699745 merged by jenkins-bot:

[translatewiki@master] repong: Add backport command

https://gerrit.wikimedia.org/r/699745

Change 716214 had a related patch set uploaded (by Nikerabbit; author: Nikerabbit):

[translatewiki@master] repomulti: add support for backport command

https://gerrit.wikimedia.org/r/716214

Change 720949 had a related patch set uploaded (by Nikerabbit; author: Nikerabbit):

[translatewiki@master] Commit support for backports

https://gerrit.wikimedia.org/r/720949

Change 720959 had a related patch set uploaded (by Nikerabbit; author: Nikerabbit):

[mediawiki/extensions/Translate@master] Avoid duplicated directory separators for premade groups

https://gerrit.wikimedia.org/r/720959

Change 720968 had a related patch set uploaded (by Nikerabbit; author: Nikerabbit):

[mediawiki/extensions/Translate@master] Backports: Preserve order of message keys

https://gerrit.wikimedia.org/r/720968

Change 720959 merged by jenkins-bot:

[mediawiki/extensions/Translate@master] Avoid duplicated directory separators for premade groups

https://gerrit.wikimedia.org/r/720959

Change 716214 merged by jenkins-bot:

[translatewiki@master] repomulti: add support for backport command

https://gerrit.wikimedia.org/r/716214

Change 720968 merged by jenkins-bot:

[mediawiki/extensions/Translate@master] Backports: Preserve order of message keys

https://gerrit.wikimedia.org/r/720968

Nikerabbit changed the task status from Open to In Progress.Sat, Sep 25, 8:56 AM