@CMyrick-WMF in the wiki dataset, which ISO code should we provide, alpha-2 or alpha-3? The language dataset should have both, but for the wiki dataset we should just pick one to serve as the "foreign key" to the language dataset.

I would guess that the alpha-3 is the best option, as alpha-2 seems like it won't cover everything. Theoretically, we could have one field that contains the alpha-2 if it exists and the alpha-3 otherwise, but that seems messy and confusing.

Also, if you haven't already, we'll need to think about codes for multi-lingual wikis. "mul" seems the obvious choice for the incubators, but what about Wikifunctions, Commons, and Wikidata, where the content is fairly trans-lingual but the community activity is probably overwhelming English?

in the wiki dataset, which ISO code should we provide, alpha-2 or alpha-3? The language dataset should have both, but for the wiki dataset we should just pick one to serve as the "foreign key" to the language dataset.
I would guess that the alpha-3 is the best option, as alpha-2 seems like it won't cover everything.

Yes, I agree that the alpha-3 code makes the most sense for a foreign key, since not all languages have an alpha-2 and all languages should have an alpha-3.

Theoretically, we could have one field that contains the alpha-2 if it exists and the alpha-3 otherwise, but that seems messy and confusing.

^ Relatedly, it could be helpful to have a "Wikimedia language code" which -- for the most part, but of course with edge cases -- maps to the alpha-2 if it exists and alpha-3 otherwise. This could potentially also serve as the foreign key. But I realize, too, that this Wikimedia language code is technically already available in the TSV via domain_name or database_code with some REGEX.

Also, if you haven't already, we'll need to think about codes for multi-lingual wikis. "mul" seems the obvious choice for the incubators, but what about Wikifunctions, Commons, and Wikidata, where the content is fairly trans-lingual but the community activity is probably overwhelming English?

Yes, I agree "mul" makes the most sense, per Wikipedia and SIL.

@CMyrick-WMF thank you! That's very helpful.

In T346855#10613358, @CMyrick-WMF wrote:

^ Relatedly, it could be helpful to have a "Wikimedia language code" which -- for the most part, but of course with edge cases -- maps to the alpha-2 if it exists and alpha-3 otherwise. This could potentially also serve as the foreign key. But I realize, too, that this Wikimedia language code is technically already available in the TSV via domain_name or database_code with some REGEX.

I definitely agree! I also agree the database code isn't enough since one of the points of this is to save folks from having to mess around with regex to get this basic information. But the great thing is we actually already have this! 😁 It's the language_code field (description in the readme).

Ideally it would be named wikimedia_language_code instead, but I think we're stuck with the existing name, at least for now!

We actually already have this! 😁 It's the language_code field

OH! 🤦🏻‍♀️ Thanks for reminding me.

@CMyrick-WMF I just realized you've actually been working on T392951: Create a first version of the canonical language dataset! We've actually decided not to do this task (add ISO 639 codes to the wiki dataset), since we're just going to use the existing Wikimedia language code as the foreign key to the new language dataset.

I'll move the status over to the other task and decline this one.