Page MenuHomePhabricator

Set up a CirrusSearch cluster in codfw (Dallas, Texas)
Closed, ResolvedPublic

Description

We want to have a functioning Search cluster in codfw. We assumed we want an AP system, so we'll keep the two clusters decoupled and what will happen is:

  • Any Cirrus job is enqueued and writes to both DCs [1]
  • If a job on one DC fails, re-enqueue just that job

[1] How to do this is debatable: if we do the parsing once and just make the jobrunners in the primary dc talk to the ElasticSearch cluster, we spare quite a few resources, but we have an higher network traffic. If we spawn a job on the secondary DC jobqueue instead, it will be a bit more complex to manage and we use more resources, but we will save network bandwidth. T105705 is related to this.

Apart from design decisions, the steps here will be:

  • Procure the hardware - 24 of the nicest servers we have in eqiad for search (?)
  • Set up the hardware in mutliple rows/racks
  • Maybe throw in 3 small/old spares as master-only nodes?
  • Puppet - check the puppet code for ''eqiadisms''
  • Actually implement the job changes to write to both datacenters.

Event Timeline

Joe created this task.Jul 13 2015, 4:23 PM
Joe raised the priority of this task from to Needs Triage.
Joe updated the task description. (Show Details)
Joe added projects: acl*sre-team, Discovery.
Joe added a subscriber: Joe.
Restricted Application added subscribers: Matanya, Aklapper. · View Herald TranscriptJul 13 2015, 4:23 PM
Joe updated the task description. (Show Details)Jul 13 2015, 4:45 PM
Joe set Security to None.
Joe triaged this task as High priority.Jul 14 2015, 7:06 AM
Deskana renamed this task from Cirrus search in codfw to Set up a CirrusSearch cluster in codfw (Dallas, Texas).Sep 15 2015, 4:26 PM
chasemp renamed this task from Set up a CirrusSearch cluster in codfw (Dallas, Texas) to [EPIC] Set up a CirrusSearch cluster in codfw (Dallas, Texas).Sep 18 2015, 2:38 PM
chasemp lowered the priority of this task from High to Normal.
chasemp added a subscriber: chasemp.Nov 9 2015, 5:07 PM

ping'd on irc but, is this epic task now done? anything remaining?

Deskana closed this task as Resolved.Nov 9 2015, 5:45 PM
Deskana claimed this task.
Deskana added a subscriber: Deskana.

ping'd on irc but, is this epic task now done? anything remaining?

As far as I know. it's resolved. Yay! :-)

EBernhardson added a subscriber: EBernhardson.

This is now up and running with a full copy of the index and all writes going to it. We should do a load test and ensure this meets our expectations before declaring victory though.

chasemp reopened this task as Open.Nov 10 2015, 4:03 PM

reopening for the remaining subtask :)

faidon renamed this task from [EPIC] Set up a CirrusSearch cluster in codfw (Dallas, Texas) to Set up a CirrusSearch cluster in codfw (Dallas, Texas).Jan 19 2016, 3:25 PM
faidon closed this task as Resolved.
faidon added a subscriber: faidon.

We already have two tasks tracking a real world load testing (T117714 & T121741) and this task's name is a bit misleading since we have set up the codfw cluster for some time now. Resolving this one.