@Trizek-WMF FYI?

We just met this morning to sort out our timeline -- current plan is to do the do the upgrade on Nov 16. That means the disturbance will start then, and should finish on or about Nov 23. (I'll ask around, but I think there's no way to get a better estimate of how long the maintenance script will take to chew through enwiki, besides "well, probably about the same as last time." We might be able to get more precise once it's running, but that won't help you for Tech News.)

It's not impossible that start date will slip by a day or two, but I think it's very likely we'll hit it, and it should be safe to announce.

The last time was T189486 in 2018:

Announcements went out in Tech/News (13 and 15) and to the Village Pumps of the largest affected communities, using this message https://www.mediawiki.org/wiki/Wikimedia_Technical_Operations/ICU_announcement

I'm going to add something to Tech News, and we could refine it next week.

Thank you for finding the old task and the message! <3

I've updated the message with the version, the start date and the Phab ticket. Based on this task description, it seems to be the only changes required. However, could you double-check? Thank you. :)

• https://meta.wikimedia.org/wiki/Tech/News/2020/46
• https://meta.wikimedia.org/wiki/Tech/News/2020/47

Yep, that text looks good to me -- the "eight of the ten biggest Wikipedias" language is the last thing I wanted to verify, and it's still correct (jawiki and zhwiki are the exceptions). If you get any questions like "is my wiki affected?" you can find the full list at T264991#6609917. Any wiki not listed there should experience no disruption with the upgrade.

Thank you for the review and this list. I asked volunteers for translations and I will message the most impacted wikis on Friday.

I preventively posted a message on English Wikipedia, since they will be the most impacted.

I've been blocked by a last minute change made on translation, which required me to manually change date formats in translations. Also, creating a distribution list matching the list of targeted wiki to their village pumps was much more time consuming than I anticipated.

The message has now been distributed.

We've started upgrading the canary appservers to ICU 63, so the window of category sorting disruption has officially started.

All appservers are now running ICU 63, and the collation update script is running. Earlier today should have been the moment of peak incorrectly-sorted categories, and everything should improve from here. I'll update as the script finishes.

The s5 script (shwiki, srwiki) is finished, the rest are still chugging along.

s2, s6, and s7 have also finished. The s3 worker has completed wikis up through ruwikibooks (in alphabetical order).

So far so good! :)

The good news: s3 is also finished, so only enwiki is left, taking the longest as expected.

The bad news: The enwiki script needed to be restarted due to an unexpected hiccup, so our estimated "a week or so" completion time is also restarted from today. It'll likely still be running into the Thanksgiving holiday.

Out of interest, have you been getting any inquiries about this? I don't have a great sense of whether the out-of-order category listings are actually bothering anybody.

There are some at https://en.wikipedia.org/wiki/Wikipedia:Village_pump_(technical)#Category_Navigation_is_messed_up (permalink).

I regularly check on the wikis, but I've been offline between Wednesday evening and now (I'll check the wikis after posting this message). Anyway, as of the middle of last week, I haven't seen many discussions. All of them were not worth being reported, since other users pointed out to the announcement concerning this maintenance operation, which solved the topic. Overall, and fortulately, our users are patient and understanding. :)

I'll leave a message at English Wikipedia. @RLazarus, what was this hiccup you mention? People may like to know about the details.

In T267145#6640181, @Trizek-WMF wrote:

@RLazarus, what was this hiccup you mention? People may like to know about the details.

The Data Persistence team needed to restart a database replica for routine work. Most of our production software can just retry any particular query on a different DB host, so that isn't usually disruptive. But updateCollation.php (the script that's migrating categories from the old collation to the new one) is only rarely used for this particular maneuver and hasn't had all that much engineering effort invested to make it resilient to that sort of event - it just connects to one replica and uses it for the entire run. When we got unlucky and that replica was restarted, instead of just reconnecting to a different host, the script exited (with no resume capability built in).

Obviously all that would be high on our list of things to improve for next time, but we're instead planning on reworking this whole process so that this user-visible migration period doesn't exist. We fully expect there won't be a "next time" at all.

This is finished on all wikis, a little bit slower than our original schedule but not stretching into the holiday weekend as I'd feared. :)

If there are any sorting issues that still persist, those are unexpected, and users should please open bug reports as they normally would. @Trizek-WMF I'm leaving this open in case there's any post-work you want to do, but it's finished from our perspective -- close it at your convenience. Thanks again for all the help!