Many language wiki templates (pl, it, en, cs) don't accept xx-XX style language codes
Open, LowPublic0 Story Points

Description

IT wiki:
You can see it here.
(I also put the same citations at en.wiki in case it's useful or interesting for you to see the different outcome.)
A solution to a similar issue is discussed in https://phabricator.wikimedia.org/T97256#1248815 .

PL wiki:
https://www.mediawiki.org/w/index.php?title=Topic:Sgikbv81nxsv7oy6&topic_showPostId=sqt0thxnx0gx69xc#flow-post-sqt0thxnx0gx69xc

EN wiki:
VE is setting the language in {{cite}} templates. It's setting it to en-US, en-GB and other flavours which are not recognized by {{cite}}. Also, these shouldn't be set if the language and wiki are the same languages. On enwiki, these errors end up in [[Category:CS1 maint: Unrecognized language]]

CS wiki:
Per community discussion here, spotted also with another problem described in T156548

Examples are:
https://en.wikipedia.org/w/index.php?title=Chris_Harris_(Automotive_Journalist)&action=edit&oldid=688324259
https://en.wikipedia.org/w/index.php?title=Adam_Waito&type=revision&diff=685374219&oldid=685373324
https://en.wikipedia.org/w/index.php?title=Aijia&type=revision&diff=681819929&oldid=681818020

Elitre created this task.Oct 13 2015, 2:29 PM
Elitre updated the task description. (Show Details)
Elitre raised the priority of this task from to Needs Triage.
Elitre added a project: Citoid.
Elitre added a subscriber: Elitre.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 13 2015, 2:29 PM
Elitre updated the task description. (Show Details)Oct 13 2015, 2:29 PM
Elitre set Security to None.
Elitre added a subscriber: Mvolz.
Mvolz added a subscriber: mobrovac.Nov 3 2015, 2:46 PM

So, this is basically the result of us now scraping more data. Our language validator has always allowed xx-XX style language codes, we just weren't getting them as often so it wasn't as noticeable.

We're currently not entirely certain how to resolve this; each template has its own way of validating language codes, and we don't want to overfit to a particular template. We'd like to conform to a given standard but are not sure what that would be. We're currently basically using https://en.wikipedia.org/wiki/IETF_language_tag (as noted by @mobrovac in chat) but not very strictly.

Mvolz renamed this task from "Unknown language" error on it.wp for sources in Italian to Many language wiki templates (pl, it, en) don't accept xx-XX style language codes.Nov 3 2015, 2:49 PM
Mvolz updated the task description. (Show Details)
Mvolz updated the task description. (Show Details)
Elitre added a comment.Nov 5 2015, 6:51 PM

@Mvolz, so who can help you to move this forward, anything I can do here? Is "not scraping more data until a fix is found" a possible solution? Any advice we can give to communities to "fix" this on their side if possible, other than the workaround linked above? Thank you!

Restricted Application added a subscriber: StudiesWorld. · View Herald TranscriptNov 5 2015, 6:51 PM
Mvolz added a comment.Nov 30 2015, 9:03 AM

@Elitre, re moving things forward- I think we are basically still undecided on what to do.

A possible fix, which is safe probably for most templates, is to stick to two-three letter language codes, but there have been complaints about the existing language codes being too limiting. But that's something I'm willing to do- @mobrovac?

Mvolz triaged this task as Normal priority.Nov 30 2015, 9:03 AM
Mvolz claimed this task.

A possible fix, which is safe probably for most templates, is to stick to two-three letter language codes, but there have been complaints about the existing language codes being too limiting.

Let's pick a standard and enforce it?

I think we agree on that, just which standard, is the issue.

nl.wp doesn't recognize xx-XX languages codes like nl-NL, the language templates accept only two letter codes (sometimes three letter codes)

Mvolz moved this task from Backlog to IO Tasks on the Citoid board.Jan 12 2016, 10:14 AM
Mvolz removed Mvolz as the assignee of this task.Sep 30 2016, 2:39 PM
Restricted Application added a project: VisualEditor. · View Herald TranscriptSep 30 2016, 2:39 PM
Jdforrester-WMF lowered the priority of this task from Normal to Low.Oct 4 2016, 7:14 PM
Jdforrester-WMF added a subscriber: Jdforrester-WMF.

So, the options are:

  • Ignore this.
  • Modify the citoid service to send less information, except magically when it's wanted (like pt-BR vs. pt); no idea how we'd all agree on a shared set for everyone.
  • The above, but inside the Citoid extension, so all clients of the service would have to replicate the same logic (but more flexible to adjust on a per-wiki basis).
  • Fix the templates to work with these valid codes.

Or am I missing something? Option 4 seems the obvious winner…

Stryn added a comment.Oct 5 2016, 9:46 AM

On fiwiki we have {{IETF-kielisymboli}} that converts "en-EN" to show like "en" would show. And if the site language is also Finnish, then this won't show it. This is using the codeLangue3 function in Module:FrLangue.

German WP has no problems at all with any language code.

If we are told explicitly that a book is written in German we store this information, but don't show that in articles and do not bother readers, but expose it in microformats.

May I advertise Multilingual lua library, e,g, getBase function? It falls back to root language for those who cannot deal with extended codes right now. Later it may be configured to support variants unknown to CLDR. Publications written in multiple languages are supported, too.

Mvolz moved this task from IO Tasks to Zotero & External on the Citoid board.Oct 28 2016, 3:24 PM
Mvolz renamed this task from Many language wiki templates (pl, it, en) don't accept xx-XX style language codes to Many language wiki templates (pl, it, en, cs) don't accept xx-XX style language codes.
Mvolz added a subscriber: Dvorapa.
Dvorapa updated the task description. (Show Details)Jan 28 2017, 1:36 PM

Really low priority?

JAnD added a subscriber: JAnD.Jan 30 2017, 6:38 AM
Jdforrester-WMF set the point value for this task to 0.Feb 9 2017, 6:14 PM
Nnvu added a subscriber: Nnvu.Apr 10 2017, 1:15 PM