I agree this must be done, and should also be submitted upstream to CLDR.

The suggested order is probably OK, but I cross-checked it with the Academic Dictionary of the Bashkir Language (Ufa 2011):

... and I have a few questions:

• Should Й be between И and К?
• It looks like Ң and Ҫ can never be in the beginning of the word. It's probably not an issue, and they should be sorted as the initial task description suggests, but I'm checking just in case :)
• Ъ, Ы, and Ь are not mentioned at all. Judging by the image in the dictionary and by Wikipedia articles such as Ыҡ, Ы can even appear in the beginning of the word. I guess they should be between Щ and Э. Again, just checking.

Amire80, thanks for your comment and your attention.
Answers from the Bashkir native (User:З. ӘЙЛЕ):
“1. Й - Yes, it should, because some names beginning from the letter Й.

2. Yes, you're right.
3. Yes, there are few words beginning from Ы. And even a pair of surnames. If we need to set the sorting strictly according to the Bashkir alphabet, then let's do it. Surely, some letters will be “empty”, but I think that's okay.”

OK, this clarifies everything I need to know, and I support doing this of course, but I wonder how exactly. May @Nemo_bis, @matmarex, or @Nikerabbit will know the precise process:

• Eventually this should be in ICU. But how does it get there? If I submit this as a ticket to CLDR, will it eventually get also to ICU?
• Until it gets into ICU, can we change it locally in MediaWiki?

I couldn't find the answers in mediawiki.org. It should be documented in https://www.mediawiki.org/wiki/Localisation .

(Added the word "collation" to the task description so it would be easier to find.)

I'm not really that familiar with how the process works. I only worked on implementing MediaWiki support for languages that were already supported by ICU.

Looking at commenters on a similar task T48235: Kurdish Wikipedia: Alphabetical order in the categories (collation), @Bawolff or @kaldari might be able to help.

We can't easily add collations unsupported by ICU in MediaWiki. The best thing we could do is to use a collation for a similar language instead, if the alphabet is close enough.

Thanks @matmarex. I see that the Kurdish issue, which you cited, involves a CLDR ticket as I thought, so I submitted one for Bashkir: http://unicode.org/cldr/trac/ticket/10195

Suggestions about anything else that can be done to make this faster or better are welcome :)

As far as I remember, there was a problem with sorting of the letter Ё in Russian Wikipedia, but it was a long time ago and I don't know how this problem was fixed.

Yes, getting it into cldr is the first step. Be prepared for this being a super-long time before the change gets to wikimedia.

For the cldr ticket, it may help to clarify if there are any secondary differences in the sorting i.e. things that only come into play in the event of a tie - e.g. diacritics. It would also be good to specify which things in the ordering are only tertiary differences (e.g Case differences). Specifying the proposed sort order in the ldml tailoring notation would probably be ideal (http://www.unicode.org/reports/tr35/tr35-collation.html#Rules ) [im guessing here. Im not associated with cldr so i really have no idea what they actually want. This is just a guess]-

We can try to hack something together in the meantime for mediawiki (unfortunately php bindings to libicu dont support tailoring, so the most easiest way ofdoing that is impossible, but we can still replace letters and things).

In T162823#3182417, @Amire80 wrote:

I couldn't find the answers in mediawiki.org. It should be documented in https://www.mediawiki.org/wiki/Localisation .

Tips to get a locale added to CLDR are available at https://translatewiki.net/wiki/CLDR .

Change 350792 had a related patch set uploaded (by Brian Wolff; owner: Brian Wolff):
[mediawiki/core@master] Add collation for Bashkir (ba)

https://gerrit.wikimedia.org/r/350792

Change 350792 merged by jenkins-bot:
[mediawiki/core@master] Add collation for Bashkir (ba)

https://gerrit.wikimedia.org/r/350792

Now that we've written the workaround, we need a configuration change to set $wgCategoryCollation = 'uppercase-ba' for the Bashkir Wikipedia, and confirmation from the community that it behaves like they expect.

Change 353099 had a related patch set uploaded (by Amire80; owner: Amire80):
[operations/mediawiki-config@master] Set collation for Bashkir wikis to uppercase-ba

https://gerrit.wikimedia.org/r/353099

Here are good page for testing after deployment:

• https://ba.wikipedia.org/wiki/%D0%9A%D0%B0%D1%82%D0%B5%D0%B3%D0%BE%D1%80%D0%B8%D1%8F:%D0%A5%D3%99%D1%80%D0%B5%D1%84%D1%82%D3%99%D1%80
• https://ba.wikipedia.org/wiki/%D0%9A%D0%B0%D1%82%D0%B5%D0%B3%D0%BE%D1%80%D0%B8%D1%8F:%D0%A0%D3%99%D1%81%D3%99%D0%B9_%D1%81%D1%83%D0%B1%D1%8A%D0%B5%D0%BA%D1%82%D1%82%D0%B0%D1%80%D1%8B_%D0%B1%D0%B0%D1%88_%D2%A1%D0%B0%D0%BB%D0%B0%D0%BB%D0%B0%D1%80%D1%8B

Change 353099 merged by jenkins-bot:
[operations/mediawiki-config@master] Set collation for Bashkir wikis to uppercase-ba

https://gerrit.wikimedia.org/r/353099

Mentioned in SAL (#wikimedia-operations) [2017-06-12T13:13:50Z] <hashar@tin> Synchronized wmf-config/InitialiseSettings.php: Set collation for Bashkir wikis to uppercase-ba - T162823 (duration: 00m 41s)

In T162823#3340526, @hashar wrote:

But fawiki has 9 left over links:

I guess you meant bawiki :)

9 rows in set (0.00 sec)

[bawiki]>

Mmmm... is there anything left to do? Does it have to remain open?

Why wasn't it completely done the first time?

I would like one to figure out why there are 9 entries in categorylinks still having cl_collation='uppercase' (instead of uppercase-ba). The break down being:

[bawiki]> SELECT cl_type,cl_collation,count(*) FROM categorylinks GROUP BY cl_type,cl_collation;
+---------+--------------+----------+
| cl_type | cl_collation | count(*) |
+---------+--------------+----------+
| page    | uppercase    |        3 |
| page    | uppercase-ba |   304451 |
| subcat  | uppercase    |        6 |
| subcat  | uppercase-ba |    59154 |
| file    | uppercase-ba |     2944 |
+---------+--------------+----------+
5 rows in set (0.24 sec)
[bawiki]>

Is this query from labs replica or actual db? In the past labs replicas have had replication issues related to DELETEs on categorylinks table that cause it to retain old rows that arent really there

The cl_from entries in those pages are missing (i.e. no page table entry) e.g. https://ba.wikipedia.org/w/index.php?curid=122102&uselang=en gives a badtitle error.

Category pages inner join on the page table, so these entries wont be shown on categories

That is from the production database. Since the pages no more exist (thank you for the check), I guess they are just artifact entry and we don(t have any system to garbage collect them.