Run `php maintenance/updateCollation.php --force` on all Russian-language projects using uca-ru collation
Closed, ResolvedPublic1 Estimated Story Points
Actions

Assigned To

Authored By

	matmarex
	Mar 9 2016, 9:48 PM

Description

Please run php maintenance/updateCollation.php --force on all Russian-language projects using uca-ru collation. For some reason the sortkeys generated before January 2015 and those generated later use a different format, resulting in incorrect ordering (see T88088: Incorrect sorting in categories on Russian-language projects).

I think we're planning to upgrade our version of ICU soon, which will require running this script on these wikis anyway (T86096), so if that's happening within next few weeks, we could probably do it only once later.

Related Objects
Search...

Status	Subtype	Assigned	Task
Resolved		kaldari	T128483 Fix category headers for pages that begin with numbers
Resolved		kaldari	T8948 Natural number sorting in category listings
Resolved		None	T88088 Incorrect sorting in categories on Russian-language projects
Resolved		Joe	T129411 Run `php maintenance/updateCollation.php --force` on all Russian-language projects using uca-ru collation
Resolved		Joe	T86096 Switch HAT appservers to trusty's ICU (or newer)
Resolved		Joe	T78765 Convert jobrunners to HHVM
Resolved		Joe	T84842 Convert eqiad imagescalers to HHVM, Trusty
Resolved		tstarling	T91468 HHVM with FastCGI does not support streaming output
Resolved		Joe	T93194 Create an HHVM 3.6.0 package, adding Tim's streaming patch
Resolved		kaldari	T58041 updateCollation.php script prohibitively slow for very large wikis
Resolved		• jcrespo	T130692 Add new indexes from eec016ece6d2b30addcdf3d3efcc2ba59b10e858 to production databases
Resolved		Volans	T128353 Switchover to new s3 master
Resolved	PRODUCTION ERROR	aaron	T126436 Spikes of mediawiki in read only for job runners after altering the s2 slaves topology
Invalid		None	T126632 Scap should restart job runners to pick up new config

Event Timeline

matmarex created this task.Mar 9 2016, 9:48 PM

Restricted Application added subscribers: JEumerus, Matanya, Aklapper. · View Herald TranscriptMar 9 2016, 9:48 PM

matmarex added a parent task: T88088: Incorrect sorting in categories on Russian-language projects.Mar 9 2016, 9:50 PM

matmarex mentioned this in T88088: Incorrect sorting in categories on Russian-language projects.

matmarex updated the task description. (Show Details)Mar 9 2016, 9:54 PM

@kaldari Mentioning you since you folks at Community Tech are currently doing category collation stuff.

@faidon Do you know if T86096 is happening soon?

Dereckson moved this task from Backlog to Analysis / under discussion on the Wikimedia-Site-requests board.Mar 9 2016, 11:52 PM

Wiki	select count(*) from categorylinks
ruwiki	12100754
ruwikibooks	11322
ruwikinews	250614
ruwikiquote	64954
ruwikisource	2138959
ruwikiversity	21140
ruwikivoyage	15483
ruwiktionary	9302371

I scheduled ruwikibooks, ruwikivoyage, ruwikiversity, ruwikinews, ruwikiquote script runs for today's evening SWAT (they should take less than 15 minutes collectively).

ruwiki, ruwikisource and ruwiktionary should probably wait for T58041 at least.

matmarex triaged this task as Medium priority.Mar 14 2016, 7:21 PM

[00:46] <MaxSem> !log ran mwscript maintenance/updateCollation.php --wiki=ruwikibooks --force
[00:51] <MaxSem> !log ran mwscript maintenance/updateCollation.php --wiki=ruwikivoyage --force
[00:52] <MaxSem> !log ran mwscript maintenance/updateCollation.php --wiki=ruwikiversity --force
[00:55] <MaxSem> !log ran mwscript maintenance/updateCollation.php --wiki=ruwikiquote --force
[01:01] <MaxSem> !log ran mwscript maintenance/updateCollation.php --wiki=ruwikinews --force

matmarex updated the task description. (Show Details)Mar 15 2016, 12:06 AM

matmarex added a subtask: T58041: updateCollation.php script prohibitively slow for very large wikis.Mar 17 2016, 1:26 AM

matmarex mentioned this in T130692: Add new indexes from eec016ece6d2b30addcdf3d3efcc2ba59b10e858 to production databases.Mar 23 2016, 1:03 AM

• jcrespo closed subtask T58041: updateCollation.php script prohibitively slow for very large wikis as Resolved.May 3 2016, 1:51 PM

• jcrespo reopened subtask T58041: updateCollation.php script prohibitively slow for very large wikis as Open.

The new index is in place on ruwiktionary now. Let's try running the updateCollation.php script there and see how fast it goes.

kaldari added a project: Community-Tech-Sprint.May 3 2016, 5:06 PM

kaldari set the point value for this task to 1.

• DannyH added a parent task: T8948: Natural number sorting in category listings.May 3 2016, 5:07 PM

@jcrespo, @matmarex: I'm currently running the updateCollation.php script for ruwiktionary, but it seems extremely slow (although it's supposed to have the new index). It's been running for 20 minutes and it's only processed 4200 rows out of 9+ million:

kaldari@terbium:/srv/mediawiki/php-1.27.0-wmf.22$ mwscript maintenance/updateCollation.php --wiki=ruwiktionary --force
...
Selecting next 100 rows... processing...4200 done.

At this rate it will take 30 days to finish running :( Any suggestions?

batch size should probably be set via a parameter

I stopped the script after 60,000 rows had been processed. It had slowed down even more and at the rate it was going it would have taken 66 days to finish. I guess for some reason our new index didn't actually help in this case.