Page MenuHomePhabricator
Create Task
Maniphest T348370

Language code for Wikifunctions natural language objects mixes different types of codes
Closed, ResolvedPublic
Actions

Description

The "code" field for natural language objects in Wikifunctions mixes different types of codes. Some are MediaWiki language codes, some are BCP 47 language codes. These are not fully compatible with each other, which makes it unclear how they should be interpreted, and in some cases, unclear what is meant.

MediaWiki contains mappings from its language codes to BCP 47-compatible ones in https://gerrit.wikimedia.org/g/mediawiki/core/+/master/includes/language/LanguageCode.php. The codes in DEPRECATED_LANGUAGE_CODE_MAPPING plus egl are also MediaWiki languages, the rest are generally only used to create valid HTML lang attributes and not otherwise understood by MediaWiki.

Examples:

https://www.wikifunctions.org/view/de/Z1624

  • Code: ar-001
  • Note: ar-001 is a BCP 47-compatible code used by CLDR but not MediaWiki.

https://www.wikifunctions.org/view/de/Z1124

  • Code: en-x-simple
  • Code aliases: simple
  • Note: The MediaWiki code is simple. The equivalent BCP 47 code is en-simple. en-x-simple is a BCP 47-compatible code using a private use subtag.

https://www.wikifunctions.org/view/de/Z1679

  • Code: roa-tara
  • Note: roa-tara is a MediaWiki code without an assigned BCP 47 code.

https://www.wikifunctions.org/view/de/Z1782

  • Code: cbk-zam
  • Note: cbk-zam is a MediaWiki code without an assigned BCP 47 code.

https://www.wikifunctions.org/view/de/Z1826

  • Code: map-bms
  • Note: map-bms is a MediaWiki code without an assigned BCP 47 code.

https://www.wikifunctions.org/view/de/Z1503

  • Code: bho
  • Code aliases: bh
  • Note: bho is the BCP 47 code and preferred MediaWiki code. bh is a MediaWiki code which means Bihari in BCP 47.

https://www.wikifunctions.org/view/de/Z1346

https://www.wikifunctions.org/view/en/Z1229

  • Code: zh-classical
  • Note: zh-classical is a deprecated MediaWiki code. The BCP 47 code and preferred MediaWiki code is lzh (= Z1152).
  • Note: Duplication solved on Wikifunctions, cf. Z1152 & Z1229 discussion

https://www.wikifunctions.org/view/de/Z1849

  • Code: nrm
  • Note: This is a MediaWiki code. The equivalent BCP 47 code is nrf. nrm means Narom in BCP 47.

Details

SubjectRepoBranchLines +/-
Update function-schemata sub-module to HEAD (ed0033d)mediawiki/extensions/WikiLambdamaster+1 -1
wikifunctions: Upgrade evaluators from 2024-02-26-150300 to 2024-04-03-210033operations/deployment-chartsmaster+2 -2
wikifunctions: Upgrade orchestrator from 2024-03-05-140533 to 2024-04-04-132719operations/deployment-chartsmaster+1 -1
Update function-schemata sub-module to HEAD (97ea4cd)mediawiki/extensions/WikiLambdamaster+5 -5
Customize query in gerrit
TitleReferenceAuthorSource BranchDest Branch
Update function-schemata sub-module to HEAD (86cfa0b)repos/abstract-wiki/wikifunctions/function-evaluator!215jforrestersync-function-schematamain
Update function-schemata sub-module to HEAD (86cfa0b)repos/abstract-wiki/wikifunctions/function-orchestrator!171apinesync-function-schematamain
Update function-schemata sub-module to HEAD (97ea4cd)repos/abstract-wiki/wikifunctions/wikilambda-cli!33jforrestersync-function-schematamain
Update function-schemata sub-module to HEAD (97ea4cd)repos/abstract-wiki/wikifunctions/function-orchestrator!142jforrestersync-function-schematamain
Update function-schemata sub-module to HEAD (97ea4cd)repos/abstract-wiki/wikifunctions/function-evaluator!179jforrestersync-function-schematamain
definitions: Fold Z1229/zh-classical into Z1152/lzh, delete the formerrepos/abstract-wiki/wikifunctions/function-schemata!98wsungdev-98main
definitions: Update nan/nan-*repos/abstract-wiki/wikifunctions/function-schemata!97wsungdev-97main
Customize query in GitLab

Related Objects

Event Timeline

Nikki created this task.Oct 7 2023, 8:30 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 7 2023, 8:30 AM
Maintenance_bot added a project: Abstract Wikipedia team.Oct 7 2023, 8:46 AM
Jdforrester-WMF moved this task from To triage to Needs Product/Design on the Abstract Wikipedia team board.Oct 12 2023, 4:39 PM
Quiddity updated the task description. (Show Details)Oct 18 2023, 7:59 PM
Quiddity subscribed.
Quiddity updated the task description. (Show Details)Oct 18 2023, 8:06 PM
Bugreporter added a project: Language codes.Dec 25 2023, 4:25 AM
Winston_Sung subscribed.Jan 27 2024, 7:54 PM
Winston_Sung updated the task description. (Show Details)Jan 28 2024, 7:27 AM
Winston_Sung updated the task description. (Show Details)Feb 25 2024, 6:51 PM
CodeReviewBot added a project: Patch-For-Review.Feb 25 2024, 7:35 PM

wsung opened https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-schemata/-/merge_requests/97

definitions: Fold Z1501/zh-min-nan into Z1221/nan, delete the former

Winston_Sung added a comment.Feb 25 2024, 7:38 PM

Merge request 97 had a related commit uploaded (by wsung; author: wsung):

[abstract-wiki/wikifunctions/function-schemata/@main] definitions: Fold Z1501/zh-min-nan into Z1221/nan, delete the former

https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-schemata/-/merge_requests/97

CodeReviewBot added a comment.Feb 25 2024, 7:45 PM

wsung opened https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-schemata/-/merge_requests/98

definitions: Fold Z1229/zh-classical into Z1152/lzh, delete the former

CodeReviewBot added a comment.Feb 26 2024, 3:02 PM

jforrester updated https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-schemata/-/merge_requests/98

definitions: Fold Z1229/zh-classical into Z1152/lzh, delete the former

CodeReviewBot added a comment.Feb 26 2024, 3:05 PM

jforrester merged https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-schemata/-/merge_requests/98

definitions: Fold Z1229/zh-classical into Z1152/lzh, delete the former

gerritbot added a comment.Mar 5 2024, 3:43 PM

Change 1002570 had a related patch set uploaded (by Jforrester; author: Jforrester):

[mediawiki/extensions/WikiLambda@master] Update function-schemata sub-module to HEAD (97ea4cd)

https://gerrit.wikimedia.org/r/1002570

gerritbot added a comment.Mar 5 2024, 8:34 PM

Change 1002570 merged by jenkins-bot:

[mediawiki/extensions/WikiLambda@master] Update function-schemata sub-module to HEAD (97ea4cd)

https://gerrit.wikimedia.org/r/1002570

DVrandecic assigned this task to Jdforrester-WMF.Mar 21 2024, 5:29 PM
Jdforrester-WMF updated the task description. (Show Details)Mar 21 2024, 5:46 PM
Jdforrester-WMF added a comment.Mar 21 2024, 5:53 PM

I've gone through this bit of a disaster of a ticket and split out the actionable items into their own tasks. Yet again, please do not create messes like this that don't get resolved and instead confuse issues together.

Of the remaining items:

  • Yes, we have BCP47-compatible tags that aren't supported by MW but we support. I don't think there's anything to action?
    • ar-001
    • en-x-simple
  • Yes, we have potentially BCP47-incompatible tags that MW supports; until there's a BCP47 tag to make primary, I don't think there's anything to action?
    • roa-tara
    • cbk-zam
    • map-bms
  • I don't know which nrf people think MW's nrm should be merged into, be it Z1169/nrf-gg or Z1337/nrf-je.

If that covers everything, we can close this task and make progress on the individual suggestions rather than being blocked.

Bugreporter subscribed.Mar 22 2024, 1:18 AM

Yes, we have potentially BCP47-incompatible tags that MW supports

In https://gerrit.wikimedia.org/g/mediawiki/core/+/3b94ef0e93553e4440acb9236cb3459fc192270a/includes/language/LanguageCode.php#81:

  • roa-tara is mapped to nap-x-tara
  • map-bms is mapped to jv-x-bms
  • cbk-zam is mapped to cbk
  • nrm is mapped to nrf

In my opinion we should use these mapped BCP47-compatible codes only.

gerritbot added a comment.Apr 4 2024, 1:43 PM

Change #1017060 had a related patch set uploaded (by Jforrester; author: Jforrester):

[operations/deployment-charts@master] wikifunctions: Upgrade orchestrator from 2024-03-05-140533 to 2024-04-04-132719

https://gerrit.wikimedia.org/r/1017060

gerritbot added a comment.Apr 4 2024, 1:44 PM

Change #1017061 had a related patch set uploaded (by Jforrester; author: Jforrester):

[operations/deployment-charts@master] wikifunctions: Upgrade evaluators from 2024-02-26-150300 to 2024-04-03-210033

https://gerrit.wikimedia.org/r/1017061

gerritbot added a comment.Apr 4 2024, 4:00 PM

Change #1017060 merged by jenkins-bot:

[operations/deployment-charts@master] wikifunctions: Upgrade orchestrator from 2024-03-05-140533 to 2024-04-04-132719

https://gerrit.wikimedia.org/r/1017060

gerritbot added a comment.Apr 4 2024, 4:06 PM

Change #1017061 merged by jenkins-bot:

[operations/deployment-charts@master] wikifunctions: Upgrade evaluators from 2024-02-26-150300 to 2024-04-03-210033

https://gerrit.wikimedia.org/r/1017061

Jdforrester-WMF moved this task from Needs Product/Design to Backlog on the Abstract Wikipedia team board.Apr 11 2024, 3:42 PM
CodeReviewBot added a comment.Mon, May 13, 9:52 PM

jforrester updated https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-schemata/-/merge_requests/97

definitions: Update nan/nan-*

CodeReviewBot added a comment.Mon, May 13, 9:54 PM

jforrester merged https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-schemata/-/merge_requests/97

definitions: Update nan/nan-*

gerritbot added a comment.Mon, May 13, 9:55 PM

Change #1031090 had a related patch set uploaded (by Jforrester; author: Jforrester):

[mediawiki/extensions/WikiLambda@master] Update function-schemata sub-module to HEAD (ed0033d)

https://gerrit.wikimedia.org/r/1031090

gerritbot added a comment.Tue, May 14, 3:25 PM

Change #1031090 merged by jenkins-bot:

[mediawiki/extensions/WikiLambda@master] Update function-schemata sub-module to HEAD (ed0033d)

https://gerrit.wikimedia.org/r/1031090

Jdforrester-WMF closed this task as Resolved.Tue, May 14, 4:00 PM

OK, at this point I believe each of the individual proposals has its own task, some of which are done. Let's create tasks for these in future.

ReleaseTaggerBot added a project: MW-1.43-notes (1.43.0-wmf.6; 2024-05-21).Tue, May 14, 4:00 PM
Winston_Sung updated the task description. (Show Details)Tue, May 14, 6:03 PM
CodeReviewBot added a comment.Tue, May 14, 6:15 PM

wsung updated https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-schemata/-/merge_requests/98

definitions: Fold Z1229/zh-classical into Z1152/lzh, delete the former

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL