I tested it on test.wikidata.org too and it seems that it uses whatever you type if you don't explicitly select an option. For example, if I type "English" and submit the form, it uses "English" as the language code. On top of that, filtering only seems to work if you type a language code, so typing "hr" will highlight Croatian, but typing "Croatian" does nothing. If you do explicitly select something, it only displays the language code which also might encourage people to try to "fix" it by entering the language name instead.

It seems I also can't remove the bad terms, when I tried to delete the "Türkçe" description on https://www.wikidata.org/wiki/Q6052351, I get "Unrecognized value for parameter 'language': Türkçe"

Change 296198 had a related patch set uploaded (by Addshore):
Don't allow bad lang codes in SpecialNewEntity

https://gerrit.wikimedia.org/r/296198

The possible damage is shown on https://www.wikidata.org/wiki/User:Pasleim/Language_statistics_for_items.

In T138725#2411058, @Sjoerddebruin wrote:

The possible damage is shown on https://www.wikidata.org/wiki/User:Pasleim/Language_statistics_for_items.

Once the fix is in place these should be easily fixable using wbeditentity (I think).

Change 296198 merged by jenkins-bot:
Don't allow bad lang codes in SpecialNewEntity

https://gerrit.wikimedia.org/r/296198

Does this still need a backport? Or can this be closed as resolved?

@thiemowmde depending on when this would normally get deployed a backport would make sense.
We should also discuss cleaning up the bad data (making sure it is possible)

Is is also impossible to delete such value - see https://www.wikidata.org/wiki/Q208693 and try to remove "aln espaniol" one, you'd get Unrecognized value for parameter 'language': aln espaniol. I think deletion should be possible in this case.

https://www.wikidata.org/wiki/User:Pasleim/Language_statistics_for_items indicates we have a bunch of bad items, too bad it doesn't have Q-ids for items related to those strings, it's be then easier to get some bot to clean them up. I can try to make a tool scanning the dump for them though.

I had a look at the wbsetlabel API module, but it's not possible to have API parameters that depend on each other. The only solution would be to set the language parameter to string, with no strict limitation to a set of known languages, and do the strict validation manually whenever a label is set.

Same must then be done for all API modules.

I suggest to not do this because it makes the code error prone and is not necessary after a proper cleanup.

But removing the data may still be possible using wbeditentity?

I'll go and take a look now!

It is possible to remove the bad languages using wgeditentity

https://www.wikidata.org/w/index.php?title=Q208693&diff=356535483&oldid=355910545

1. Get the whole entity with wbgetentities
2. Remove the bad language codes from the JSON
3. Send the whole data back to wbeditentity with the CLEAR parameter set!

I've cleaned up most of the languages that were obviously bad. Didn't touch ones like aeb-latn, but if anybody knows how to easily get the list of allowed languages it should be pretty easy to run a script to clean ones with non-matching langs.

Just to get this on the record on the task:
What is needed with this issue now? Is it still UBN!? Should follow-up occur in a separate task?

I think the UBN part has been fixed, as bad language codes are not added anymore and bad data has been cleaned up from wikidata.