Page MenuHomePhabricator

Access translation strings via JavaScript
Closed, ResolvedPublic

Description

The Flask-Babel tool we are using does not supply any native way to access translations via JavaScript, so would require hacky solutions like adding data attributes or hidden elements to the HTML.

We have several areas of the application where we must use text in JavaScript. The DataTables plugin for example allows you to provide strings to use for the buttons it generates, but they must be defined in JS.

Outline

String extraction from JavaScript files

The main GNU gettext library allows extraction from JavaScript (and all other main languages). This can be merged into the existing messages.pot file, before continuing with the usual compiling process to reach a .mo file.

NOTE: The API does not extract from HTML, or any HTML template format. Hence it's best to leave the existing Flask-Babel extraction to deal with the Python and HTML, and only use GNU gettext API for JavaScript.
Displaying language content in JavaScript files

Use the i18n Webpack Plugin to compile different language versions of each JavaScript file automatically.
So instead of outputting each file to /static/js folder, we output to /static/js/<lang code>/ instead.
There will be 1 copy of each output JavaScript file per language (initially, that means one full set of js files in /static/js/en and one full set in /static/js/fr)

In the source JavaScript files (src folder), we can mark a translatable string using the format __('some translatable text').

The i18n-webpack-plugin then replaces this this text with actual strings for each language. This happens when webpack compiles the the output files (during npm run build or npm run dev).

i18n-webpack-plugin needs a .json file to find the translations, so we use po2json to create a json conversion of each language's .po file.

Finally, the <script> tag used on each page loading our JavaScript files needs to be updated to use the session language to determine which /static/js/<lang code> folder to load from. I.e. we only load from the js file for the current language

Instructions

String Extraction

These are the steps that need to be taken in addition to the existing extracting/compiling:
For all steps you should be in the isa subfolder within the main project folder

  1. Make sure you're messages.pot file is ready and up to date using Flask-Babel. At this stage you have all translation strings extracted from Python and HTML template files
  1. Generate list of JavaScript files in the .src folder. The list is stored in a file, so you do not need to repeat next time unless the list of JS files changes.
find src/*.js -type f -print > js-file-list
  1. Invoke the xgettext command using the generated list as input, using --join-existing to add extracted translation to the existing .pot file
xgettext --files-from=js-file-list --output=messages.pot --language=JavaScript --from-code=UTF-8 --join-existing --omit-header

Now the messages.pot file contains all original strings, plus the new JavaScript strings that have been merged in

  1. Continue with the normal workflow with Flask-Babel and generate the .po files for each language (or update the .po files if they already exist). The .po files will then need the actual translations added for the new JavaScript strings
  1. Convert each .po file to JSON using po2json and save it as messages.json (in same folder as the .po and .mo file)

This has been setup as an npm script to build French json file (the only one we need initially).

npm run build-fr-json
Usage in files
  1. In any of the source js files, use the format __('some translatable text')

This can then be extracted to the .pot file in the process described above.
It will also be automatically replaced with the relevant translation for each language version of the js file.

IMPORTANT: i18n-webpack plugin does not have built in support for plural strings, or sprintf expressions. For the next improvement, we can use sprintf-js for inserting variables into strings, but this would need a different .po to .json conversion - the current po2json does not preserve the required format %{variable name}. Alternatively, a solution like jed-webpack-plugin would allow everything in one package, including parsing .mo files directly. Unfortunately this does not seem to work with latest webpack version.

Event Timeline

NavinoEvans triaged this task as Normal priority.Sun, Jul 28, 11:23 AM
NavinoEvans created this task.
Eugene233 rescinded a token.
Eugene233 awarded a token.
NavinoEvans moved this task from Pending to In progress on the ISA board.Thu, Aug 1, 11:35 AM
NavinoEvans updated the task description. (Show Details)Thu, Aug 1, 2:37 PM
NavinoEvans updated the task description. (Show Details)
NavinoEvans updated the task description. (Show Details)Sun, Aug 4, 3:00 PM
NavinoEvans updated the task description. (Show Details)Mon, Aug 5, 9:34 AM
NavinoEvans updated the task description. (Show Details)Mon, Aug 5, 9:49 AM
NavinoEvans updated the task description. (Show Details)Tue, Aug 6, 11:48 AM
NavinoEvans updated the task description. (Show Details)
NavinoEvans updated the task description. (Show Details)
NavinoEvans updated the task description. (Show Details)
NavinoEvans updated the task description. (Show Details)
NavinoEvans updated the task description. (Show Details)Tue, Aug 6, 11:50 AM
NavinoEvans updated the task description. (Show Details)Thu, Aug 8, 5:46 PM
NavinoEvans updated the task description. (Show Details)Fri, Aug 9, 10:52 AM
NavinoEvans updated the task description. (Show Details)Fri, Aug 9, 11:06 AM
NavinoEvans updated the task description. (Show Details)Fri, Aug 9, 11:15 PM
NavinoEvans updated the task description. (Show Details)Sun, Aug 11, 1:13 PM
NavinoEvans updated the task description. (Show Details)
NavinoEvans updated the task description. (Show Details)Wed, Aug 14, 8:20 PM
NavinoEvans updated the task description. (Show Details)Wed, Aug 14, 8:46 PM
NavinoEvans updated the task description. (Show Details)Wed, Aug 14, 9:19 PM
NavinoEvans updated the task description. (Show Details)Wed, Aug 14, 9:52 PM

Change 530238 had a related patch set uploaded (by NavinoEvans; owner: NavinoEvans):
[labs/tools/Isa@master] Access translation strings via JavaScript

https://gerrit.wikimedia.org/r/530238

NavinoEvans moved this task from In progress to Review on the ISA board.Thu, Aug 15, 10:21 AM

Change 530238 merged by jenkins-bot:
[labs/tools/Isa@master] Access translation strings via JavaScript

https://gerrit.wikimedia.org/r/530238

Eugene233 closed this task as Resolved.Mon, Aug 19, 10:03 AM
Eugene233 added a subscriber: Eugene233.

Marking as resolved as T230646 requires that we close this task to be sure that ISA is i18n-ready.