Page MenuHomePhabricator

Move all apihelp translatable messages to separate files
Open, Needs TriagePublic

Description

I'd like to start this by saying that translatable apihelp messages are great.

Some people think that all people who write software know English, and don't understand why should they be translated at all. This notion is incorrect. I heard from numerous people that they use the API for developing gadgets, and who happily use the translations of Apihelp in Special:ApiSandbox into their language—it's easier for them than to read the API documentation in English. So thanks, @Anomie, for making it possible.

That said, these messages are probably read by fewer people than other user interface messages. Hence, it should be a lower priority to translate them than to translate other messages in core and extensions, which are actually seen by end users who read and edit the wikis. Apihelp messages are useful only to the developers and there are fewer developers than editors.

To actually help translators at translatewiki to do this, Apihelp messages should be separated from other user interface messages. This was already done for core MediaWiki, as well as for several extensions: UniversalLanguageSelector, Translate, ContentTranslation, Echo, Flow, and FlaggedRevisions.

This should be done for all extensions. In particular, this should be done for Wikibase, which has hundreds of apihelp messages, but there are many other extensions that need this.

This requires careful and substantial work, but the process is well-known and there are existing examples. I am therefore marking it as good first bug. This involves:

  • In the extension's Git repository:
    • Under the extension's i18n/ directory, create the api/ directory.
    • In this directory, create en.json and qqq.json files.
    • Find API-related messages. Usually their keys begin with apihelp-*, apierror-*, or apiwarn-*
    • Copy all the API messages from en.json and qqq.json in the i18n/ directory to en.json and qqq.json in the i18n/api/ directory
    • Delete the API messages from the original json files.
    • If the extension has extension.json, edit this file and add "i18n/api" to MessagesDirs.
    • If the extension has no extension.json, edit its main PHP initialization file (called ExtensionName.php) and add the api directory to MessagesDirs. See the example from CirrusSearch.
    • If the extension has Gruntfile.js and it has a banana section, then:
      • If you edited extension.json earlier, edit Gruntfile.js in a way that reads MessageDirs from extension.json. See the example from AbuseFilter.
      • If there is no extensions.json, add api: 'i18n/api/' in the banana section.
  • In the translatewiki Git repository:
    • Split the extension configuration in these files:
      • groups/MediaWiki/mediawiki-extensions.txt
      • groups/MediaWiki/ExtensionsAgg.yaml
      • groups/MediaWiki/Wikimedia*Agg.yaml (could be WikimediaMainAgg.yaml, WikimediaAdvancedAgg.yaml, etc.)
    • Notes:
      • The new API group name will be "ext-extensionname-api".
      • The group with the messages for users will be "ext-extensionname-user"
      • Reuse the extension's description message for the aggregate group's description field. The description message usually has the name "extensionname-desc". For example, for TimedMediaHandler the description value in the YAML file is: description: '{{Special:MyLanguage/MediaWiki:Timedmediahandler-desc/en}}'. Please check that this message actually exists on translatewiki by going to the page MediaWiki://extensionname//-desc/en (example).
      • Create a new aggregate group "ext-extensionname", including the main extension and the API messages using "ext-extensionname-*"
      • Put the API messages group to groups/MediaWiki/WikimediaTechnicalAgg.yaml.
      • Change the entry of the old message group in the aggregate groups files to the new group name. For example, if "ext-scribunto" appeared in WikimediaTechnicalAgg.yaml, the entry must be changed to "ext-scribunto-user".
      • If the previously defined message group has optional or ignored messages and api* messages appear there, move them to the new api group.

Examples:

This is it. Existing translations will be migrated automatically.

See more links to patches in the checklist toward the end of the task description.

The impact for the extensions' developers will be that they'll have to add new apihelp/apierror/apiwarn messages to the i18n/api/en.json and i18n/api/qqq.json files, instead of i18n/en.json and i18n/qqq.json. As far as I can tell, nobody complained about having to do this with Echo, Flow, Translate, etc.

The impact for the translators will be that the cryptic, highly technical, and rarely-seen translatable messages in the "Extensions Used by Wikimedia" group will be separated. As of August 2018, there are more than 1900 such API messages.


Checklist:

This checklist was written semi-manually. It is probably not complete, but it does include some of the larger and more important extensions. Once many of these are done, some more will be added.

If you are working on converting an extension, put your username in the parentheses after its name. Projects marked as (+) are up for grabs, and highly recommended if you're looking for an extension to send a simple patch for because they are among the more commonly translated extensions.


A somewhat related task: T167762: Split core en.json to several files.

Related Objects

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Amire80 updated the task description. (Show Details)May 11 2019, 8:15 AM

Change 509572 had a related patch set uploaded (by Amire80; owner: Amire80):
[mediawiki/extensions/SiteMatrix@master] Split apihelp messages to a separate file

https://gerrit.wikimedia.org/r/509572

Change 509573 had a related patch set uploaded (by Amire80; owner: Amire80):
[translatewiki@master] Split api messages from SiteMatrix

https://gerrit.wikimedia.org/r/509573

Amire80 updated the task description. (Show Details)May 11 2019, 8:34 AM

Change 509574 had a related patch set uploaded (by Amire80; owner: Amire80):
[translatewiki@master] Split api messages from CategoryTree

https://gerrit.wikimedia.org/r/509574

Change 509575 had a related patch set uploaded (by Amire80; owner: Amire80):
[mediawiki/extensions/CentralNotice@master] Split apihelp messages to a separate file

https://gerrit.wikimedia.org/r/509575

Amire80 updated the task description. (Show Details)May 11 2019, 8:51 AM
Zoranzoki21 updated the task description. (Show Details)May 16 2019, 7:42 PM
Zoranzoki21 added a subscriber: MarcoAurelio.
Zoranzoki21 updated the task description. (Show Details)May 16 2019, 7:46 PM
Zoranzoki21 updated the task description. (Show Details)May 16 2019, 7:49 PM

Change 499292 merged by jenkins-bot:
[mediawiki/extensions/Newsletter@master] Split API messages to a separate file

https://gerrit.wikimedia.org/r/499292

Change 499286 merged by jenkins-bot:
[translatewiki@master] [Newsletter] Split API messages to a separate file

https://gerrit.wikimedia.org/r/499286

Amire80 updated the task description. (Show Details)Jun 1 2019, 7:18 AM
Amire80 added subscribers: Zoranzoki21, MarcoAurelio.

Change 509575 merged by jenkins-bot:
[mediawiki/extensions/CentralNotice@master] Split apihelp messages to a separate file

https://gerrit.wikimedia.org/r/509575

Change 509574 merged by jenkins-bot:
[translatewiki@master] Split api messages from CentralNotice

https://gerrit.wikimedia.org/r/509574

Amire80 updated the task description. (Show Details)Jul 3 2019, 12:00 PM

Change 520731 had a related patch set uploaded (by Amire80; owner: Amire80):
[mediawiki/extensions/OATHAuth@master] Split apihelp messages to a separate file

https://gerrit.wikimedia.org/r/520731

Amire80 updated the task description. (Show Details)Jul 4 2019, 11:55 AM

Change 520743 had a related patch set uploaded (by Amire80; owner: Amire80):
[mediawiki/extensions/Linter@master] Split apihelp messages to a separate file

https://gerrit.wikimedia.org/r/520743

Change 520744 had a related patch set uploaded (by Amire80; owner: Amire80):
[translatewiki@master] Split api messages from OATHAuth

https://gerrit.wikimedia.org/r/520744

Change 520748 had a related patch set uploaded (by Amire80; owner: Amire80):
[translatewiki@master] Split api messages from Linter

https://gerrit.wikimedia.org/r/520748

Amire80 updated the task description. (Show Details)Jul 4 2019, 12:45 PM

Change 520731 merged by jenkins-bot:
[mediawiki/extensions/OATHAuth@master] Split apihelp messages to a separate file

https://gerrit.wikimedia.org/r/520731

Change 509572 merged by jenkins-bot:
[mediawiki/extensions/SiteMatrix@master] Split apihelp messages to a separate file

https://gerrit.wikimedia.org/r/509572

Change 509569 merged by jenkins-bot:
[mediawiki/extensions/CategoryTree@master] Split apihelp messages to a separate file

https://gerrit.wikimedia.org/r/509569

Change 520743 merged by jenkins-bot:
[mediawiki/extensions/Linter@master] Split apihelp messages to a separate file

https://gerrit.wikimedia.org/r/520743

Change 520748 merged by jenkins-bot:
[translatewiki@master] Split api messages from Linter

https://gerrit.wikimedia.org/r/520748

Change 509573 merged by jenkins-bot:
[translatewiki@master] Split api messages from SiteMatrix

https://gerrit.wikimedia.org/r/509573

Change 520744 merged by jenkins-bot:
[translatewiki@master] Split api messages from OATHAuth

https://gerrit.wikimedia.org/r/520744

Change 509570 merged by jenkins-bot:
[translatewiki@master] Split api messages from CategoryTree

https://gerrit.wikimedia.org/r/509570

Amire80 updated the task description. (Show Details)Jul 5 2019, 8:59 PM

Change 521340 had a related patch set uploaded (by Amire80; owner: Amire80):
[mediawiki/extensions/Babel@master] Split apihelp messages to a separate file

https://gerrit.wikimedia.org/r/521340

Change 521341 had a related patch set uploaded (by Amire80; owner: Amire80):
[translatewiki@master] Split api messages from Babel

https://gerrit.wikimedia.org/r/521341

Change 521342 had a related patch set uploaded (by Amire80; owner: Amire80):
[mediawiki/extensions/FeaturedFeeds@master] Split apihelp messages to a separate file

https://gerrit.wikimedia.org/r/521342

Change 521343 had a related patch set uploaded (by Amire80; owner: Amire80):
[translatewiki@master] Split api messages from FeaturedFeeds

https://gerrit.wikimedia.org/r/521343

Amire80 updated the task description. (Show Details)Jul 9 2019, 11:22 AM

Change 521340 merged by jenkins-bot:
[mediawiki/extensions/Babel@master] Split apihelp messages to a separate file

https://gerrit.wikimedia.org/r/521340

Change 521341 merged by jenkins-bot:
[translatewiki@master] Split api messages from Babel

https://gerrit.wikimedia.org/r/521341

Change 521680 had a related patch set uploaded (by Amire80; owner: Amire80):
[translatewiki@master] Split api messages from Kartographer and add WV to aggregate group

https://gerrit.wikimedia.org/r/521680

Change 521682 had a related patch set uploaded (by Amire80; owner: Amire80):
[mediawiki/extensions/Kartographer@master] Split apihelp messages to a separate file

https://gerrit.wikimedia.org/r/521682

Amire80 updated the task description. (Show Details)Jul 9 2019, 9:45 PM

Change 522109 had a related patch set uploaded (by Amire80; owner: Amire80):
[mediawiki/extensions/BetaFeatures@master] Split apihelp messages to a separate file

https://gerrit.wikimedia.org/r/522109

Change 522114 had a related patch set uploaded (by Amire80; owner: Amire80):
[translatewiki@master] Split api messages from BetaFeatures

https://gerrit.wikimedia.org/r/522114

Change 522116 had a related patch set uploaded (by Amire80; owner: Amire80):
[mediawiki/extensions/Cite@master] Split apihelp messages to a separate file

https://gerrit.wikimedia.org/r/522116

Change 522117 had a related patch set uploaded (by Amire80; owner: Amire80):
[translatewiki@master] Split api messages from Cite

https://gerrit.wikimedia.org/r/522117

Amire80 updated the task description. (Show Details)Jul 11 2019, 3:24 PM
Amire80 updated the task description. (Show Details)Jul 11 2019, 9:14 PM

Change 521342 merged by jenkins-bot:
[mediawiki/extensions/FeaturedFeeds@master] Split apihelp messages to a separate file

https://gerrit.wikimedia.org/r/521342

Change 521682 merged by jenkins-bot:
[mediawiki/extensions/Kartographer@master] Split apihelp messages to a separate file

https://gerrit.wikimedia.org/r/521682

Change 522109 merged by jenkins-bot:
[mediawiki/extensions/BetaFeatures@master] Split apihelp messages to a separate file

https://gerrit.wikimedia.org/r/522109

Change 522116 merged by jenkins-bot:
[mediawiki/extensions/Cite@master] Split apihelp messages to a separate file

https://gerrit.wikimedia.org/r/522116

Change 521343 merged by jenkins-bot:
[translatewiki@master] Split api messages from FeaturedFeeds

https://gerrit.wikimedia.org/r/521343

Change 521680 merged by jenkins-bot:
[translatewiki@master] Split api messages from Kartographer and add WV to aggregate group

https://gerrit.wikimedia.org/r/521680

Change 522114 merged by jenkins-bot:
[translatewiki@master] Split api messages from BetaFeatures

https://gerrit.wikimedia.org/r/522114

Change 522117 merged by jenkins-bot:
[translatewiki@master] Split api messages from Cite

https://gerrit.wikimedia.org/r/522117

Base added a subscriber: Base.Sun, Aug 11, 2:45 PM
Krenair updated the task description. (Show Details)Sun, Aug 11, 3:29 PM
Krenair added a subscriber: Krenair.EditedSun, Aug 11, 4:35 PM

I made a script to automate the moving around of JSON stuff, suggest this is not run with a version of Python below 3.6 due to dictionary ordering:

import json, os
os.makedirs(os.path.join('i18n', 'api'), exist_ok=True)
with open('i18n/en.json') as englishMainFile:
    englishMainData = json.load(englishMainFile)

with open('i18n/qqq.json') as docsMainFile:
    docsMainData = json.load(docsMainFile)

englishApiData = {"@metadata": {"authors": englishMainData.get("@metadata", {}).get("authors", [])}}
docsApiData = {"@metadata": {"authors": docsMainData.get("@metadata", {}).get("authors", [])}}
for key in list(englishMainData):
    if key.startswith('api'):
        englishApiData[key] = englishMainData[key]
        docsApiData[key] = docsMainData[key]
        del englishMainData[key]
        del docsMainData[key]

with open('i18n/api/en.json', 'w') as englishApiFile:
    json.dump(englishApiData, englishApiFile, indent='\t')

with open('i18n/api/qqq.json', 'w') as docsApiFile:
    json.dump(docsApiData, docsApiFile, indent='\t')

with open('i18n/en.json', 'w') as englishMainFile:
    json.dump(englishMainData, englishMainFile, indent='\t')

with open('i18n/qqq.json', 'w') as docsMainFile:
    json.dump(docsMainData, docsMainFile, indent='\t')

if os.path.exists('extension.json'):
    with open('extension.json') as extensionFile:
        extensionData = json.load(extensionFile)

    msgDir, = extensionData['MessagesDirs'].keys()
    extensionData['MessagesDirs'][msgDir].append('i18n/api')

    with open('extension.json', 'w') as extensionFile:
        json.dump(extensionData, extensionFile, indent='\t')

    if os.path.exists('Gruntfile.js'):
        print('If Gruntfile.js has a banana section, edit it to read MessageDirs from extension.json. See https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/AbuseFilter/+/420371/12/Gruntfile.js for an example')
else:
    print('Edit the main PHP initialisation file and add the api directory to MessagesDirs. See https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/CirrusSearch/+/474164/4/CirrusSearch.php for an example')

    if os.path.exists('Gruntfile.js'):
        print("If Gruntfile.js has a banana section, add `api: 'i18n/api'` to it.")

Have not fully tested it yet.

Change 529610 had a related patch set uploaded (by Alex Monk; owner: Alex Monk):
[mediawiki/extensions/BounceHandler@master] Split apihelp messages to a separate file

https://gerrit.wikimedia.org/r/529610

Krenair added a comment.EditedSun, Aug 11, 11:26 PM

Experimental code to produce TWN changes
As with the above, this is all hacked together just as a one-off task to generate commits that will themselves be reviewed, not intended for long-term maintenance or anything

import json, ruamel.yaml, sys
extension = sys.argv[1]
yaml = ruamel.yaml.YAML()
yaml.indent(mapping=2, sequence=4, offset=2)

with open('../MediaWiki/extensions/BounceHandler/extension.json') as f:
    description_msg = json.load(f)['descriptionmsg']

description_msg = description_msg[0].upper() + description_msg[1:]
id = 'ext-{}'.format(extension.lower())

fname = 'groups/MediaWiki/WikimediaAdvancedAgg.yaml'
d = ruamel.yaml.YAML().load(open(fname))
group_idx = d['GROUPS'].index(id)
d['GROUPS'][group_idx] = '{}-user'.format(id)
yaml.dump(d, open(fname, 'w'))

fname = 'groups/MediaWiki/WikimediaTechnicalAgg.yaml'
d = ruamel.yaml.YAML().load(open(fname))
groups_sorted = sorted(d['GROUPS'] + [id])
group_sorted_idx = groups_sorted.index(id)
sorted_predecessor = groups_sorted[group_sorted_idx - 1]
predecessor_idx = d['GROUPS'].index(sorted_predecessor)
d['GROUPS'].insert(predecessor_idx + 1, '{}-api'.format(id))
yaml.dump(d, open(fname, 'w'))

fname = 'groups/MediaWiki/ExtensionsAgg.yaml'
docs = list(ruamel.yaml.YAML().load_all(open(fname)))
ids = [id] + [doc['BASIC']['id'] for doc in docs if 'BASIC' in doc]
ids.sort()
sorted_idx = ids.index(id)
sorted_predecessor = ids[sorted_idx - 1]
with open(fname, 'w') as f:
    with ruamel.yaml.YAML(output=f) as yamlf:
        yamlf.indent(mapping=2, sequence=4, offset=2)
        for i, doc in enumerate(docs):
            yamlf.dump(doc)
            if i < len(docs) - 1:
                f.write('\n')

            if 'BASIC' in doc and doc['BASIC']['id'] == sorted_predecessor:
                yamlf.dump({
                    'BASIC': {
                        'id': id,
                        'label': extension,
                        'description': '{{Special:MyLanguage/MediaWiki:' + description_msg + '/en}}'
                    },
                    'GROUPS': [
                        '{}-*'.format(id)
                    ]
                })
                f.write('\n')

sections = {}
outLines = []
gotHeader = False
existingUI = False
with open('groups/MediaWiki/mediawiki-extensions.txt') as f:
    for i, line in enumerate(f.read().splitlines()):
        if line.startswith('file = {}/i18n/%CODE%.json'.format(extension)):
            outLines[-1] += ' - User Interface'
            outLines.append('id = {}-user'.format(id))
            existingUI = True
        if line == '':
            gotHeader = True
        elif gotHeader:
            sections[line] = i
            gotHeader = False
        outLines.append(line)

if not existingUI:
    addHeader = '{} - User Interface'.format(extension)
    addLines = [addHeader, 'id = {}-user'.format(id), 'file = {}/i18n/%CODE%.json'.format(extension), '']

    sortedSections = sorted(list(sections.keys()) + [addHeader])
    successor_sorted = sortedSections[sortedSections.index(addHeader) + 1]
    successor_line_no = sections[successor_sorted]
    outLines = outLines[:successor_line_no] + addLines + outLines[successor_line_no:]

addHeader = '{} - API'.format(extension)
addLines = [addHeader, 'id = {}-api'.format(id), 'file = {}/i18n/api/%CODE%.json'.format(extension), '']

sortedSections = sorted(list(sections.keys()) + [addHeader])
successor_sorted = sortedSections[sortedSections.index(addHeader) + 1]
successor_line_no = sections[successor_sorted]
outLines = outLines[:successor_line_no] + addLines + outLines[successor_line_no:]

with open('groups/MediaWiki/mediawiki-extensions.txt', 'w') as f:
    f.write('\n'.join(outLines))

Change 529623 had a related patch set uploaded (by Alex Monk; owner: Alex Monk):
[translatewiki@master] Split api messages from BounceHandler

https://gerrit.wikimedia.org/r/529623

Change 529610 merged by jenkins-bot:
[mediawiki/extensions/BounceHandler@master] Split apihelp messages to a separate file

https://gerrit.wikimedia.org/r/529610

Change 529623 merged by jenkins-bot:
[translatewiki@master] Split api messages from BounceHandler

https://gerrit.wikimedia.org/r/529623

Krenair updated the task description. (Show Details)Wed, Aug 14, 12:26 PM

Change 530118 had a related patch set uploaded (by Alex Monk; owner: Alex Monk):
[mediawiki/extensions/ExtensionDistributor@master] Split apihelp messages to a separate file

https://gerrit.wikimedia.org/r/530118

Change 530126 had a related patch set uploaded (by Alex Monk; owner: Alex Monk):
[translatewiki@master] Split apihelp messages from ExtensionDistributor

https://gerrit.wikimedia.org/r/530126

Krenair updated the task description. (Show Details)Wed, Aug 14, 1:14 PM

Change 530130 had a related patch set uploaded (by Alex Monk; owner: Alex Monk):
[translatewiki@master] Split apihelp messages from GeoData

https://gerrit.wikimedia.org/r/530130

Change 530131 had a related patch set uploaded (by Alex Monk; owner: Alex Monk):
[mediawiki/extensions/GeoData@master] Split apihelp messages to a separate file

https://gerrit.wikimedia.org/r/530131

Krenair updated the task description. (Show Details)Wed, Aug 14, 1:22 PM

Change 530141 had a related patch set uploaded (by Alex Monk; owner: Alex Monk):
[mediawiki/extensions/GettingStarted@master] Split apihelp messages to a separate file

https://gerrit.wikimedia.org/r/530141

Krenair updated the task description. (Show Details)Wed, Aug 14, 1:39 PM

Change 530142 had a related patch set uploaded (by Alex Monk; owner: Alex Monk):
[translatewiki@master] Split apihelp messages from GettingStarted

https://gerrit.wikimedia.org/r/530142

Change 530143 had a related patch set uploaded (by Alex Monk; owner: Alex Monk):
[mediawiki/extensions/GlobalBlocking@master] Split apihelp messages to a separate file

https://gerrit.wikimedia.org/r/530143

Krenair updated the task description. (Show Details)Wed, Aug 14, 1:48 PM

Change 530691 had a related patch set uploaded (by Alex Monk; owner: Alex Monk):
[translatewiki@master] Split apihelp messages from GlobalBlocking

https://gerrit.wikimedia.org/r/530691

I'd like some review before I run the script on the remaining extensions and upload the commits for those.