Page MenuHomePhabricator

Create extra elasticsearch clusters in beta cluster
Closed, ResolvedPublic

Description

We are now running three elasticsearch clusters serving cirrussearch in each datacenter for production. Beta cluster needs to be updated to run two or three clusters as well (even tiny ones) to excercise the same code paths and ensure we have a testing platform.

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 16 2019, 4:15 PM
Reedy renamed this task from Create extra elsaticsearch clusters in beta cluster to Create extra elasticsearch clusters in beta cluster.Jan 16 2019, 4:16 PM
Gehel added a project: Discovery-Search.
Gehel moved this task from needs triage to Ops / SRE on the Discovery-Search board.
jijiki triaged this task as Medium priority.Feb 7 2019, 12:47 PM

Change 497698 had a related patch set uploaded (by Mathew.onipe; owner: Mathew.onipe):
[operations/puppet@production] multi-instance for elastic deployment-prep

https://gerrit.wikimedia.org/r/497698

Change 497698 merged by Gehel:
[operations/puppet@production] multi-instance for elastic deployment-prep

https://gerrit.wikimedia.org/r/497698

Change 498937 had a related patch set uploaded (by Mathew.onipe; owner: Mathew.onipe):
[operations/puppet@production] don't change old cluster name

https://gerrit.wikimedia.org/r/498937

Change 498937 merged by Gehel:
[operations/puppet@production] don't change old cluster name

https://gerrit.wikimedia.org/r/498937

Beta cluster is running three clusters now. Cluster state is fine for all them.

chi:

onimisionipe@deployment-elastic06:~$ curl https://deployment-elastic06.deployment-prep.eqiad.wmflabs:9243/_cluster/health?pretty
{
  "cluster_name" : "beta-search",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 3,
  "number_of_data_nodes" : 3,
  "active_primary_shards" : 388,
  "active_shards" : 790,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}

omega:

onimisionipe@deployment-elastic06:~$ curl https://deployment-elastic06.deployment-prep.eqiad.wmflabs:9443/_cluster/health?pretty
{
  "cluster_name" : "beta-search-omega",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 3,
  "number_of_data_nodes" : 3,
  "active_primary_shards" : 0,
  "active_shards" : 0,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}

psi:

onimisionipe@deployment-elastic06:~$ curl https://deployment-elastic06.deployment-prep.eqiad.wmflabs:9643/_cluster/health?pretty
{
  "cluster_name" : "beta-search-psi",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 3,
  "number_of_data_nodes" : 3,
  "active_primary_shards" : 0,
  "active_shards" : 0,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
debt closed this task as Resolved.Apr 5 2019, 10:47 PM