Page MenuHomePhabricator

Defined language codes containing -x- generate errors when used for lexemes
Open, Needs TriagePublicBUG REPORT

Description

Steps to replicate the issue (include links if applicable):

Part 1:

  • Define a custom language code in LocalSettings.php which contains -x-, e.g. aa-x-localset.
  • Start adding a lexeme.
  • Select the custom language code as the spelling variant.
  • Try to create the lexeme.

Part 2:

  • Disable Javascript.
  • Start adding a lexeme.
  • Select the custom language code as the spelling variant.
  • Create the lexeme.
  • Try to edit the lemma or add a form or sense using the custom code.

What happens?:

Part 1:
The language code can be selected in the spelling variant field, but the lexeme can't be created with the error Invalid Item ID "localset" in field "lemma" at "aa-x-localset".

Part 2:
The lexeme can be created, but the lemma can't be edited, and forms and senses can't be added, with a similar error.

What should have happened instead?:

Any custom language codes defined in LocalSettings.php (or Names.php) should be allowed. The QID check should only apply to language codes not already known to MediaWiki.

While Wikimedia is not currently using codes containing -x-, codes such as map-bms and roa-tara might one day be changed to use -x-, and third party users of the extension should not be prevented from defining/adding custom BCP 47-compliant language codes.

Software version (skip for WMF-hosted wikis like Wikipedia):

Other information (browser name/version, screenshots, etc.):

Example line to add to LocalSettings.php
$wgExtraLanguageNames["aa-x-localset"] = "aa-x-localset";

Event Timeline

JS version of Special:NewLexeme:

image.png (806×472 px, 41 KB)

Errors if you create the lexeme with the no-JS version of Special:NewLexeme and then try to edit it (I’ve pasted the screenshots together in Gimp):
T325728.png (971×800 px, 87 KB)

Now that it uses language codes from the CLDR extension, this will presumably also happen if any codes containing -x- are added there.

Some things to bear in mind:

BCP47-compatible language tags may only contain one x prefix - it signals the start of private-use subtags, it's not a prefix for each private-use subtag. Therefore, if you want to add a QID to a language tag which already contains -x-, it needs to be added to the end without another -x-, e.g. roa-x-tararoa-x-tara-Q12345

BCP47-compatible language tags can also start with x- to form entirely private-use tags (e.g. the x-whatever example on page 81 of the RFC). If someone were to define one for their installation, it would currently work because the code is splitting on -x-. Any changes to the validation should continue to allow them. The previous point applies here too, e.g. x-whateverx-whatever-Q12345.

A small number of BCP 47 language tags are "grandfathered" (there's a list on pages 5-6 of the RFC). They keep their meaning from older versions of the standard but can't be combined with other subtags (including private use ones), so something like zh-min-nan-x-Q12345 is not valid.

MediaWiki has a function for checking whether tags are syntactically valid (link) but unfortunately it's not much use here, because the -x-QID syntax mostly creates invalid language tags.

because the -x-QID syntax mostly creates invalid language tags.

See also: T167166: Specify the use of extended language codes in Lexemes