maintenance script to copy the ES index from one cluster to another
Closed, ResolvedPublic3 Estimated Story Points
Actions

Description

I think we can build the index in codfw and the future labs replica much quicker than a dump. Basically we just need to run the current Reindexer but with the source in one cluster and the sink in the other.

This needs to fit into the basic plan for bootstraping an additional cluster:

Create mappings for all wiki's in codfw
deploy operations/mediawiki-config to also send writes to codfw for one index
Watch things for a bit
turn on a few more. turn on the a few more. turn on the rest.
Copy/rebuild into the same index accepting writes

Additionally by having the reindexing contained under a single process we could use standard tools like trickle[1] to rate limit our WAN traffic (The Reindexer can also fork to increase the speed if needed).

[1] http://linux.die.net/man/1/trickle

Details

Subject	Repo	Branch	Lines +/-
Split connection to source and target.	mediawiki/extensions/CirrusSearch	wmf/1.27.0-wmf.2	+224 -38
Split connection to source and target.	mediawiki/extensions/CirrusSearch	master	+224 -38
Split connection to source and target.	mediawiki/extensions/CirrusSearch	wmf/1.27.0-wmf.3	+224 -38

Customize query in gerrit

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Resolved		• Deskana	T105703 Set up a CirrusSearch cluster in codfw (Dallas, Texas)
		Resolved		Smalyshev	T113018 maintenance script to copy the ES index from one cluster to another

Event Timeline

EBernhardson created this task.Sep 18 2015, 4:13 AM

EBernhardson raised the priority of this task from to Needs Triage.

EBernhardson updated the task description. (Show Details)

EBernhardson added projects: CirrusSearch, Discovery-Search (Current work).

EBernhardson added subscribers: EBernhardson, • chasemp.

Restricted Application added a project: Discovery-ARCHIVED. · View Herald TranscriptSep 18 2015, 4:13 AM

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

EBernhardson updated the task description. (Show Details)Sep 18 2015, 4:16 AM

EBernhardson set Security to None.

EBernhardson updated the task description. (Show Details)Sep 18 2015, 5:16 AM

• chasemp triaged this task as Medium priority.Sep 18 2015, 2:34 PM

• chasemp added a parent task: T105703: Set up a CirrusSearch cluster in codfw (Dallas, Texas).

updateOneSearchIndexConfig basically already does everything we need. It even takes the connection to read from (via an $index and $oldTypes constructor arguments) and the connection to write to ($connection and $types constructor arguments) independently.

updateOneSearchIndexConfig currently always creates a new index, so it isn't directly usable. Maybe a flag? maybe shared code in a new maint class? not sure yet

EBernhardson added a comment.Sep 18 2015, 6:14 PM

This comment was removed by EBernhardson.

• ksmith added a project: OKR-Work.Sep 24 2015, 4:42 PM

• ksmith moved this task from Needs triage to On Sprint Board on the Discovery-ARCHIVED board.Sep 24 2015, 5:16 PM

Smalyshev claimed this task.Oct 7 2015, 7:15 PM

Change 244247 had a related patch set uploaded (by Smalyshev):
Use corrent idnexes/connections for old and new types.

https://gerrit.wikimedia.org/r/244247

gerritbot added a project: Patch-For-Review.Oct 7 2015, 7:17 PM

EBernhardson moved this task from Incoming to not in use - please delete on the Discovery-Search (Current work) board.Oct 8 2015, 4:53 PM

Smalyshev moved this task from not in use - please delete to Needs review on the Discovery-Search (Current work) board.Oct 9 2015, 7:51 PM

Smalyshev edited a custom field.

Change 244247 merged by jenkins-bot:
Split connection to source and target.