Change Details

I think we can build the index in codfw and the future labs replica much quicker than a dump. updateOneSearchIndexConfig basically already does everything we need. It even takes the connection to read from (via an Elastica\Index object) and the connection to write to (via a CirrusSearch\Connection) independently. There is a hitch: to get a consistent index without having to manually reindex specific time periods, the plan so far has been: # Create mappings for all wiki's in codfw # deploy operations/mediawiki-config to also send writes to codfw for one index # Watch things for a bit # turn on a few more. turn on the a few more. turn on the rest. # Copy/rebuilt into the same index accepting writes updateOneSearchIndexConfig currently always creates a new index, so it isn't directly usable. Maybe a flag? maybe shared code in a new maint class? Additionally by having the reindexing contained to a single process we could use standard tools like trickle[1] to rate limit our WAN traffic. [1] http://linux.die.net/man/1/trickle

I think we can build the index in codfw and the future labs replica much quicker than a dump. updateOneSearchIndexConfig basically already does everything we need. It even takes the connection to read from (via an Elastica\Index object) and the connection to write to (via a CirrusSearch\Connection) independently. There is a hitch: to get a consistent index without having to manually reindex specific time periods, the plan so far has been: # Create mappings for all wiki's in codfw # deploy operations/mediawiki-config to also send writes to codfw for one index # Watch things for a bit # turn on a few more. turn on the a few more. turn on the rest. # Copy/rebuilt into the same index accepting writes updateOneSearchIndexConfig currently always creates a new index, so it isn't directly usable. Maybe a flag? maybe shared code in a new maint class? Additionally by having the reindexing contained under a single process we could use standard tools like trickle[1] to rate limit our WAN traffic (Reindexing can also fork to increase the speed if needed). [1] http://linux.die.net/man/1/trickle

I think we can build the index in codfw and the future labs replica much quicker than a dump. updateOneSearchIndexConfig basically already does everything we need. It even takes the connection to read from (via an Elastica\Index object) and the connection to write to (via a CirrusSearch\Connection) independently. There is a hitch: to get a consistent index without having to manually reindex specific time periods, the plan so far has been: # Create mappings for all wiki's in codfw # deploy operations/mediawiki-config to also send writes to codfw for one index # Watch things for a bit # turn on a few more. turn on the a few more. turn on the rest. # Copy/rebuilt into the same index accepting writes updateOneSearchIndexConfig currently always creates a new index, so it isn't directly usable. Maybe a flag? maybe shared code in a new maint class? Additionally by having the reindexing contained tounder a single process we could use standard tools like trickle[1] to rate limit our WAN traffic (Reindexing can also fork to increase the speed if needed). [1] http://linux.die.net/man/1/trickle