Labels in languages from $wmgExtraLanguageNames cannot be used on client wikis
Closed, ResolvedPublic
Actions

Description

As a Wikipedia editor, I want to get all labels of a Wikidata item, in order to be able to describe an item not just in the content language of my wiki, but also in several other minority languages it covers.

Problem:
Wikidata includes support for several custom language codes, by setting $wmgExtraLanguageNames in InitialiseSettings.php (warning, large file, may slow down your browser). However, those language codes are only defined for Wikidata (test and real) and Commons; on client wikis, Wikibase does not consider them to be valid, and therefore refuses to return labels in those languages, even if they’re defined on Wikidata. See discussion on Project_chat#Kven

Example:
Swedish Wikipedia would like to use the smj label of Tysfjord (Q493851), or the fkv label of Porsanger (Q483885). The following work snippets work in a Lua sandbox on Wikidata, but not in a Lua sandbox on Swedish Wikipedia:

=mw.wikibase.getLabelByLang( 'Q493851', 'smj' )
=mw.wikibase.getLabelByLang( 'Q483885', 'fkv' )

Screenshots/mockups:

BDD
GIVEN
AND
WHEN
AND
THEN
AND

Acceptance criteria:

Labels in languages from $wmgExtraLanguageNames can be used on client wikis

Open questions:

Details

Subject	Repo	Branch	Lines +/-
Remove check for valid MW language code in Lua	mediawiki/extensions/Wikibase	REL1_35	+0 -4
Remove check for valid MW language code in Lua	mediawiki/extensions/Wikibase	master	+0 -4
Make Lua language code check more lenient	mediawiki/extensions/Wikibase	master	+1 -1
Add Wikidata's $wgExtraLanguageNames to Terms WikibaseContentLanguages	mediawiki/extensions/Wikibase	master	+19 -1
Use Wikibase Term languages for SpecialItemDisambiguation	mediawiki/extensions/Wikibase	master	+1 -2

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Resolved	guergana.tzatchkova	T259340 Labels in languages from $wmgExtraLanguageNames cannot be used on client wikis
Resolved	guergana.tzatchkova	T260118 Move content of $wgExtraLanguageNames on Wikidata to default Terms languages
Stalled	Lucas_Werkmeister_WMDE	T263441 Clean up $wgExtraLanguageNames production config
Open	None	T273627 Remove wmgExtraLanguageNames from Wikimedia production
Resolved	Jakob_WMDE	T124758 [Story] Show all available languages in monolingual text value's suggester
Duplicate	None	T147839 "mul" is not listed in the suggestions for monolingual text
Duplicate	None	T273625 Make lexeme language codes inherit from Wikibase default terms languages, not MediaWiki content languages
Open	None	T320889 Use the same list of languages for monolingual text and lexemes
Open	None	T320887 Language codes that are explicitly not allowed for monolingual text should also not be allowed for lexemes
Open	None	T304494 Special:NewLexeme allows creating lexemes with spelling variants be-tarask and be-x-old

Event Timeline

Lucas_Werkmeister_WMDE created this task.Jul 31 2020, 11:06 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 31 2020, 11:06 AM

Mohammed_Sadat_WMDE subscribed.Jul 31 2020, 11:18 AM

Lydia_Pintscher added a project: Wikidata-Campsite.Jul 31 2020, 11:34 AM

Lydia_Pintscher moved this task from Incoming to Unconnected Stories on the Wikidata-Campsite board.

Mohammed_Sadat_WMDE updated the task description. (Show Details)Jul 31 2020, 11:45 AM

Dipsacus_fullonum subscribed.Jul 31 2020, 6:31 PM

Addshore moved this task from Unconnected Stories to Wikidata-Campsite-Iteration-∞ (On Hold) on the Wikidata-Campsite board.Aug 4 2020, 12:21 PM

Addshore edited projects, added Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)); removed Wikidata-Campsite.

Maintenance_bot moved this task from incoming to in progress on the Wikidata board.Aug 4 2020, 1:15 PM

Michael updated the task description. (Show Details)Aug 4 2020, 1:17 PM

guergana.tzatchkova claimed this task.Aug 10 2020, 9:36 AM

guergana.tzatchkova moved this task from To Do (prioritised from top to bottom) to Doing on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.

I'm pairing on this too!

Tarrow updated the task description. (Show Details)Aug 10 2020, 1:04 PM

Tarrow updated the task description. (Show Details)

We've decided that given the scope of this request we anticipate that the only concern is being able to use the labels in Lua. @Lydia_Pintscher does that sound ok?

A quick investigation of the property parser function seems to suggest that this particular case wouldn't be possible anyway (since it appears that only user languages can be what is rendered).

Tarrow added a subscriber: Lydia_Pintscher.Aug 10 2020, 1:18 PM

Change 619453 had a related patch set uploaded (by Tarrow; owner: Tarrow):
[mediawiki/extensions/Wikibase@master] Use Wikibase Term languages for SpecialItemDisambiguation

https://gerrit.wikimedia.org/r/619453

gerritbot added a project: Patch-For-Review.Aug 11 2020, 10:55 AM

Change 619456 had a related patch set uploaded (by Tarrow; owner: Tarrow):
[mediawiki/extensions/Wikibase@master] Add Wikidata's $wgExtraLanguageNames to Terms WikibaseContentLanguages

https://gerrit.wikimedia.org/r/619456

We've discovered that *part* of the problem here is an early escape in the lua that was added in I75d3c70302270a62db93b7e1fe790b2355f75dc2 with the intention of preventing bad keys getting into the cache.

So we also need to adapt this carefully in such a way that we don't make the cache breakable again

I guess wikibase.getLabelByLang should use isValidCode rather than isKnownLanguageTag?

Change 619725 had a related patch set uploaded (by Tarrow; owner: Tarrow):
[mediawiki/extensions/Wikibase@master] Remove check for valid MW language code in Lua

https://gerrit.wikimedia.org/r/619725

guergana.tzatchkova moved this task from Doing to Peer Review on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.Aug 12 2020, 2:14 PM

Change 619453 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Use Wikibase Term languages for SpecialItemDisambiguation

https://gerrit.wikimedia.org/r/619453

ReleaseTaggerBot added a project: MW-1.36-notes (1.36.0-wmf.5; 2020-08-18).Aug 12 2020, 5:00 PM

Change 619456 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Add Wikidata's $wgExtraLanguageNames to Terms WikibaseContentLanguages

https://gerrit.wikimedia.org/r/619456

Added the campsite for review since @guergana.tzatchkova and I aren't gonna be around on the campsite; one patch needs merging to clean up the edge case where new languages that have been added are not in cldr

Change 620996 had a related patch set uploaded (by Lucas Werkmeister (WMDE); owner: Lucas Werkmeister (WMDE)):
[mediawiki/extensions/Wikibase@master] Make Lua language code check more lenient

https://gerrit.wikimedia.org/r/620996

In T259340#6388174, @Tarrow wrote:

Added the campsite for review since @guergana.tzatchkova and I aren't gonna be around on the campsite; one patch needs merging to clean up the edge case where new languages that have been added are not in cldr

(This refers to https://gerrit.wikimedia.org/r/c/619725, but I’ve uploaded https://gerrit.wikimedia.org/r/620996 as a suggested replacement. Review from other campers appreciated!)

Change 620996 abandoned by Lucas Werkmeister (WMDE):
[mediawiki/extensions/Wikibase@master] Make Lua language code check more lenient

Reason:
in favor of I98ff5a811d

https://gerrit.wikimedia.org/r/620996

Change 619725 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Remove check for valid MW language code in Lua

https://gerrit.wikimedia.org/r/619725

ReleaseTaggerBot edited projects, added MW-1.36-notes (1.36.0-wmf.6; 2020-08-25); removed MW-1.36-notes (1.36.0-wmf.5; 2020-08-18).Aug 20 2020, 2:00 PM

Can be tested next week, I think (follow the “Lua sandbox on Swedish Wikipedia” link in the task description and paste the two code snippets into the console at the bottom).

matej_suchanek mentioned this in T261038: test_language_lists of site_tests.TestDataSiteSearchEntities fails.Aug 22 2020, 11:05 AM

Works \o/

WMDE-leszek closed subtask T260118: Move content of $wgExtraLanguageNames on Wikidata to default Terms languages as Resolved.Oct 7 2020, 3:47 PM

Change 672593 had a related patch set uploaded (by Tobias Andersson; owner: Tarrow):
[mediawiki/extensions/Wikibase@REL1_35] Remove check for valid MW language code in Lua

https://gerrit.wikimedia.org/r/672593

Change 672593 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@REL1_35] Remove check for valid MW language code in Lua