Page MenuHomePhabricator

Announcing ICU 57 transition to the community
Closed, ResolvedPublic

Description

We're about to migrate the servers running the Mediawiki application servers to a new release of the ICU Unicode library (https://en.wikipedia.org/wiki/International_Components_for_Unicode) (from 52 to 57). This unblocks some future work on upgrading the servers and will also allow the use of improved internationalization in the future (as wikis will then be able to use features introduced by the new release).

This migration will cause some unavoidable user-visible impact: The sorting of some category pages will be distorted; all pages which have been updated with the new software version
will use the new sorting while untouched pages still use the old sorting. As such, we need to run a maintenance script to update the sorting for old entries.

This affects a number of large wikis (including eight out of the ten biggest Wikipedias). The detailed list (and the task for the technical implementation) is at T189295.

The tentative date is the 9th of April 2018 (distortions may be visible for several days as the maintenance script takes quite a while to complete).

Please let me know if you need additional information.


Announcements went out in Tech/News (13 and 15) and to the Village Pumps of the largest affected communities, using this message https://www.mediawiki.org/wiki/Wikimedia_Technical_Operations/ICU_announcement

Event Timeline

https://www.mediawiki.org/wiki/User:MMuhlenhoff_(WMF)/ICU_announcement has been made, needs to be reviewed (probably simplified), marked for translation etc. Currently looking who to assign this to.

Thanks, let me know if anything else is needed on my end.

While this change will have a huge impact on the wikis, I think it should be announced in Tech News.

@MoritzMuehlenhoff, how would you summarize it in one sentence?

While this change will have a huge impact on the wikis, I think it should be announced in Tech News.

@MoritzMuehlenhoff, how would you summarize it in one sentence?

It doesn't need to be announced tomorrow though. I think we can wait for an assignee to decide the strategy and follow up here.

This unblocks some future work on upgrading the servers and will also allow the use of improved internationalization in the future (as wikis will then be able to use features introduced by the new release).

Could you describe (or link to) 1 or 2 examples of the kind of new features that will (soon or eventually) become available?

The sorting of some category pages will be distorted

IIUC, that is the only (and very temporary) problem. I don't think it is a huge impact on most editors or readers. However I'm unsure whether it will affect Tools that use the live category data? If so, beyond Tech/News and wikitech-l@, we should possibly announce to cloud@ too.

This unblocks some future work on upgrading the servers and will also allow the use of improved internationalization in the future (as wikis will then be able to use features introduced by the new release).

Could you describe (or link to) 1 or 2 examples of the kind of new features that will (soon or eventually) become available?

This change will allow us to use a more recent version of Unicode in Mediawiki (https://phabricator.wikimedia.org/T188480) and there are also new collation definitions in ICU 57 which people want to use. One example is
https://phabricator.wikimedia.org/T182431 (there might be more when digging through the ICU backlog in Phabricator).

The sorting of some category pages will be distorted

IIUC, that is the only (and very temporary) problem. I don't think it is a huge impact on most editors or readers. However I'm unsure whether it will affect Tools that use the live category data? If so, beyond Tech/News and wikitech-l@, we should possibly announce to cloud@ too.

I don't know, but I'll contact the WMCS people to find out.

While this change will have a huge impact on the wikis, I think it should be announced in Tech News.

@MoritzMuehlenhoff, how would you summarize it in one sentence?

How technical should it be? I've slightly cheated and used two sentences:

"We'll be upgrading an internationalisation library used on many wikis which will require to run a maintenance script to update existing database entries. Until the completion of the script run, the sort order of categories will be distorted."

Sounds good. We will include this in either next week's or the April 2 edition of Tech/News in the "Future changes" section. Perhaps as:

"From 9 April, the sort order of categories will be distorted for a short time. We are upgrading versions of an internationalisation library (ICU) and using a maintenance script to update existing database entries. This will last anywhere from a few hours to a few days, depending on wiki size." T189295

We can send the longer announcement text (https://www.mediawiki.org/wiki/User:MMuhlenhoff_(WMF)/ICU_announcement ) to wikitech-l@ wikitech-ambassadors@ and cloud@ on the same day.

Quiddity triaged this task as Medium priority.

"From 9 April, the sort order of categories will be distorted for a short time. We are upgrading versions of an internationalisation library (ICU) and using a maintenance script to update existing database entries. This will last anywhere from a few hours to a few days, depending on wiki size." T189295

We probably cannot run the conversion script in parallel for all affected wikis, probably only 1 or a few per shard (but that needs further discussion with our DBAs), as such I expected the accumulated time to last longer overall.

For your earlier question wrt Tools; no particular trouble with bots is known, but changes to API usage cannot be ruled either, so best to include them in the headsup I guess.

If I had to extract from last time we ran the ICU upgrade (see https://phabricator.wikimedia.org/T86096#2329554), we would have to assume a run on enwiki (132 M rows) will take around 95 hours at 1.6 million rows/hour. So I would say that we can expect the upgrade to take between 3 and 4 days, if for any reason we are getting a performance hit.

So the transition will be over on most wikis within a day, but will take 3 or more days on english wikipedia.

"From 9 April, the sort order of categories will be distorted for a short time. We are upgrading versions of an internationalisation library (ICU) and using a maintenance script to update existing database entries. This will last anywhere from a few hours to a few days, depending on wiki size." T189295

We probably cannot run the conversion script in parallel for all affected wikis, probably only 1 or a few per shard (but that needs further discussion with our DBAs), as such I expected the accumulated time to last longer overall.

This won't matter much, as enwiki has more categories of all the other wikis with ICU collation combined, and we definitely cannot run the script in parallel on a single wiki. So we can run in parallel on the other sections (in fact, we did that last time and in terms of rows affected it was quite evenly balanced).

This won't matter much, as enwiki has more categories of all the other wikis with ICU collation combined, and we definitely cannot run the script in parallel on a single wiki. So we can run in parallel on the other sections (in fact, we did that last time and in terms of rows affected it was quite evenly balanced).

For the last migration, was the updateCollation script run in parallel per shard, were there e.g. two parallel runs for ruwiki and frwiki (both in the same shard) or one script run per shard?

I've added the item to next Monday's Tech/News, and moved the more detailed description to mw:Wikimedia_Technical_Operations/ICU_announcement.
I will ask for translations of that page and also distribute the info, by sending the full text to wikitech-ambassadors@ and translators-l@, in ~24 hours.

@MoritzMuehlenhoff Just to confirm: IIUC, only Enwiki is going to take multiple days, and all other wikis will probably take less than ~12 hours from start-to-finish? If that is accurate, I'll likely not send a massmessage anywhere, but will instead just notify Enwiki's villagepump, and rely on Tech/News and mailing lists to cover everywhere else. If there are other outlier wikis that might take more than ~12 hours, could you give me a rough guess list? Thanks!

This won't matter much, as enwiki has more categories of all the other wikis with ICU collation combined, and we definitely cannot run the script in parallel on a single wiki. So we can run in parallel on the other sections (in fact, we did that last time and in terms of rows affected it was quite evenly balanced).

For the last migration, was the updateCollation script run in parallel per shard, were there e.g. two parallel runs for ruwiki and frwiki (both in the same shard) or one script run per shard?

Since the number of rows per section was comparable across the different sections, and given the activity of this script is heavy on writes, we ran 1 script per section. Overall, it lasted around 24 hours.

@MoritzMuehlenhoff Just to confirm: IIUC, only Enwiki is going to take multiple days, and all other wikis will probably take less than ~12 hours from start-to-finish? If that is accurate, I'll likely not send a massmessage anywhere, but will instead just notify Enwiki's villagepump, and rely on Tech/News and mailing lists to cover everywhere else. If there are other outlier wikis that might take more than ~12 hours, could you give me a rough guess list? Thanks!

So for the worst-affected wikis, it could extend to up to 24 hours or more - if there is some db instability we will have to turn this off and restart.

On some wikis, it will barely be noticeable for some hours, but on others it will last 12-24 hours.

Enwiki, as usual, is a beast of its own.

@MoritzMuehlenhoff Just to confirm: IIUC, only Enwiki is going to take multiple days, and all other wikis will probably take less than ~12 hours from start-to-finish? If that is accurate, I'll likely not send a massmessage anywhere, but will instead just notify Enwiki's villagepump, and rely on Tech/News and mailing lists to cover everywhere else. If there are other outlier wikis that might take more than ~12 hours, could you give me a rough guess list? Thanks!

So for the worst-affected wikis, it could extend to up to 24 hours or more - if there is some db instability we will have to turn this off and restart.

On some wikis, it will barely be noticeable for some hours, but on others it will last 12-24 hours.

Ack. Still for the announcement let's err on the side of caution and allow for some wiggle room, if it completes ahead of the announced time, all the better.

@Joe @MoritzMuehlenhoff After talking with people and looking through old tasks, I've learned that it took 7 days to do this on Enwiki last time, for T136150: Switch English Wikipedia to uca-default collation. The quantity has increased by ~14% since then, to 119,619,460 categories. I will make sure that Enwiki is prepared for a week or more of sort order quirks.

Question: Would it be relatively easy for you to generate a listing for me, of the wikis where it is very likely to take more than 1 hour? That way I could potentially do a wider targeted announcement. Thanks!

@Joe @MoritzMuehlenhoff After talking with people and looking through old tasks, I've learned that it took 7 days to do this on Enwiki last time, for T136150: Switch English Wikipedia to uca-default collation. The quantity has increased by ~14% since then, to 119,619,460 categories. I will make sure that Enwiki is prepared for a week or more of sort order quirks.

Question: Would it be relatively easy for you to generate a listing for me, of the wikis where it is very likely to take more than 1 hour? That way I could potentially do a wider targeted announcement. Thanks!

Hi, see the list at https://phabricator.wikimedia.org/T189295. Most of the wikis from s3 have few categories and while there's still quite a number of affected wikis, they probably won't need a special announcement. All the others will observe some user impact.

Ack. Still for the announcement let's err on the side of caution and allow for some wiggle room, if it completes ahead of the announced time, all the better.

Also, we need to account for the time the actual HHVM update will take; we'll need to upgrade > 120 app servers before we can start the conversion script. Since we need to avoid too many cold JIT caches we'll need to spread this out a bit.

That should be re-announced in Tech/News/2018/14. Nick will handle that. :)

Please note that since we can't run on more than one wiki in parallel per section, most wikis will complete their work within one hour, but they'll start after a long queue has completed. So expect the migration to be over in ~ 2 days on s2 and ~ 1.5 days on the other sections, with the notable exception of s1 (enwiki) - see https://phabricator.wikimedia.org/T189295#4115676 for a rough estimation of the total number of rows per section.

Notification has happened via TechNews, closing the task

(Work on this task was evaluated in Q4.)