Page MenuHomePhabricator
Create Task
Maniphest T313431

Increase Elastic master-eligible nodes from 3 to 5
Closed, ResolvedPublic2 Estimated Story Points
Actions

Description

As mentioned in T313095 , we lost a master due to hardware failure, and then a second master during a routine reimage operation.

While there was no user impact, we believe that increasing the number of Elastic master-eligible nodes from 3 to 5 will reduce the likelihood of these types of failures without significantly affecting performance or operational complexity.

We should also choose newer chassis as our master eligibles, as many of the current master-eligibles are slated for a hardware refresh

Details

Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
elasticsearch: Elasticsearch 7 does not need to specify number of mastersGeheloperations/puppetproduction+0 -9
elasticsearch: Increase number of master-eligible nodes to 5 for eqiadGeheloperations/puppetproduction+10 -4
elastic: raise master-eligibles from 3 to 5Bkingoperations/puppetproduction+6 -0
elastic: replace 2 codfw masters to be decom'dRyan Kemperoperations/puppetproduction+2 -2
Revert "elastic: reduce master-eligibles for codfw back down to 2"Bkingoperations/puppetproduction+8 -8
elasticsearch: Increase number of master-eligible nodes to 5 for codfwGeheloperations/puppetproduction+7 -7
elastic: reduce master-eligibles for codfw back down to 2Bkingoperations/puppetproduction+7 -7
elastic: Increase # of master-eligibles in codfwBkingoperations/puppetproduction+26 -20
Customize query in gerrit

Related Objects

Event Timeline

bking created this task.Jul 20 2022, 6:00 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 20 2022, 6:00 PM
Gehel moved this task from needs triage to Current work on the Discovery-Search board.Jul 25 2022, 3:37 PM
Gehel edited projects, added: Discovery-Search (Current work); removed: Discovery-Search.
• MPhamWMF set the point value for this task to 2.Aug 1 2022, 3:49 PM
• MPhamWMF moved this task from Incoming to Ready for Dev -- SWE on the Discovery-Search (Current work) board.
gerritbot added a comment.Sep 6 2022, 7:39 PM

Change 830245 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] elastic: Increase no of master-eligibles in codfw

https://gerrit.wikimedia.org/r/830245

gerritbot added a project: Patch-For-Review.Sep 6 2022, 7:39 PM
gerritbot added a comment.Sep 6 2022, 7:52 PM

Change 830245 merged by Bking:

[operations/puppet@production] elastic: Increase # of master-eligibles in codfw

https://gerrit.wikimedia.org/r/830245

Stashbot added a comment.Sep 6 2022, 8:03 PM

Mentioned in SAL (#wikimedia-operations) [2022-09-06T20:03:29Z] <inflatador> 'bking@cumin1001 disabling puppet on elastic codfw hosts T313431'

Maintenance_bot removed a project: Patch-For-Review.Sep 6 2022, 8:30 PM
gerritbot added a comment.Sep 6 2022, 8:31 PM

Change 830261 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] elastic: reduce master-eligibles for codfw back down to 2

https://gerrit.wikimedia.org/r/830261

gerritbot added a project: Patch-For-Review.Sep 6 2022, 8:31 PM
gerritbot added a comment.Sep 6 2022, 8:41 PM

Change 830261 merged by Bking:

[operations/puppet@production] elastic: reduce master-eligibles for codfw back down to 2

https://gerrit.wikimedia.org/r/830261

Maintenance_bot removed a project: Patch-For-Review.Sep 6 2022, 9:30 PM
RKemper added a comment.Sep 7 2022, 4:28 PM
ryankemper@mwmaint1002:~/elastic$ cat psi_codfw_masters.lst
elastic2054.codfw.wmnet:9700
elastic2076.codfw.wmnet:9700
elastic2080.codfw.wmnet:9700
ryankemper@mwmaint1002:~/elastic$ python push_cross_cluster_conf.py https://search.svc.codfw.wmnet:9643/_cluster/settings --ccc chi=chi_codfw_masters.lst psi=psi_codfw_masters.lst omega=omega_codfw_masters.lst
seeds=['elastic2025.codfw.wmnet:9300', 'elastic2031.codfw.wmnet:9300', 'elastic2042.codfw.wmnet:9300']
to_ret={'chi': {'seeds': ['elastic2025.codfw.wmnet:9300', 'elastic2031.codfw.wmnet:9300', 'elastic2042.codfw.wmnet:9300']}}
seeds=['elastic2054.codfw.wmnet:9700', 'elastic2076.codfw.wmnet:9700', 'elastic2080.codfw.wmnet:9700']
to_ret={'psi': {'seeds': ['elastic2054.codfw.wmnet:9700', 'elastic2076.codfw.wmnet:9700', 'elastic2080.codfw.wmnet:9700']}}
seeds=['elastic2042.codfw.wmnet:9500', 'elastic2047.codfw.wmnet:9500', 'elastic2052.codfw.wmnet:9500']
to_ret={'omega': {'seeds': ['elastic2042.codfw.wmnet:9500', 'elastic2047.codfw.wmnet:9500', 'elastic2052.codfw.wmnet:9500']}}

Set new seeds for psi to resolve elasticsearch settings alert.

(We'll need to re-do this step w/ the new new values when we expand up to 5 masters today)

gerritbot added a comment.Sep 8 2022, 7:03 PM

Change 830923 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] Revert "elastic: reduce master-eligibles for codfw back down to 2"

https://gerrit.wikimedia.org/r/830923

gerritbot added a project: Patch-For-Review.Sep 8 2022, 7:03 PM
RKemper added a comment.EditedSep 9 2022, 9:13 PM
In T313431#8218238, @RKemper wrote:
ryankemper@mwmaint1002:~/elastic$ cat psi_codfw_masters.lst
elastic2054.codfw.wmnet:9700
elastic2076.codfw.wmnet:9700
elastic2080.codfw.wmnet:9700
ryankemper@mwmaint1002:~/elastic$ python push_cross_cluster_conf.py https://search.svc.codfw.wmnet:9643/_cluster/settings --ccc chi=chi_codfw_masters.lst psi=psi_codfw_masters.lst omega=omega_codfw_masters.lst
seeds=['elastic2025.codfw.wmnet:9300', 'elastic2031.codfw.wmnet:9300', 'elastic2042.codfw.wmnet:9300']
to_ret={'chi': {'seeds': ['elastic2025.codfw.wmnet:9300', 'elastic2031.codfw.wmnet:9300', 'elastic2042.codfw.wmnet:9300']}}
seeds=['elastic2054.codfw.wmnet:9700', 'elastic2076.codfw.wmnet:9700', 'elastic2080.codfw.wmnet:9700']
to_ret={'psi': {'seeds': ['elastic2054.codfw.wmnet:9700', 'elastic2076.codfw.wmnet:9700', 'elastic2080.codfw.wmnet:9700']}}
seeds=['elastic2042.codfw.wmnet:9500', 'elastic2047.codfw.wmnet:9500', 'elastic2052.codfw.wmnet:9500']
to_ret={'omega': {'seeds': ['elastic2042.codfw.wmnet:9500', 'elastic2047.codfw.wmnet:9500', 'elastic2052.codfw.wmnet:9500']}}

Set new seeds for psi to resolve elasticsearch settings alert.

(We'll need to re-do this step w/ the new new values when we expand up to 5 masters today)

Needed to run on the other two clusters, so I just did that:

ryankemper@mwmaint1002:~/elastic$ python push_cross_cluster_conf.py https://search.svc.codfw.wmnet:9443/_cluster/settings --ccc chi=chi_codfw_masters.lst psi=psi_codfw_masters.lst omega=omega_codfw_masters.lst
seeds=['elastic2025.codfw.wmnet:9300', 'elastic2031.codfw.wmnet:9300', 'elastic2042.codfw.wmnet:9300']
to_ret={'chi': {'seeds': ['elastic2025.codfw.wmnet:9300', 'elastic2031.codfw.wmnet:9300', 'elastic2042.codfw.wmnet:9300']}}
seeds=['elastic2054.codfw.wmnet:9700', 'elastic2076.codfw.wmnet:9700', 'elastic2080.codfw.wmnet:9700']
to_ret={'psi': {'seeds': ['elastic2054.codfw.wmnet:9700', 'elastic2076.codfw.wmnet:9700', 'elastic2080.codfw.wmnet:9700']}}
seeds=['elastic2042.codfw.wmnet:9500', 'elastic2047.codfw.wmnet:9500', 'elastic2052.codfw.wmnet:9500']
to_ret={'omega': {'seeds': ['elastic2042.codfw.wmnet:9500', 'elastic2047.codfw.wmnet:9500', 'elastic2052.codfw.wmnet:9500']}}
ryankemper@mwmaint1002:~/elastic$ python push_cross_cluster_conf.py https://search.svc.codfw.wmnet:9243/_cluster/settings --ccc chi=chi_codfw_masters.lst psi=psi_codfw_masters.lst omega=omega_codfw_masters.lst
seeds=['elastic2025.codfw.wmnet:9300', 'elastic2031.codfw.wmnet:9300', 'elastic2042.codfw.wmnet:9300']
to_ret={'chi': {'seeds': ['elastic2025.codfw.wmnet:9300', 'elastic2031.codfw.wmnet:9300', 'elastic2042.codfw.wmnet:9300']}}
seeds=['elastic2054.codfw.wmnet:9700', 'elastic2076.codfw.wmnet:9700', 'elastic2080.codfw.wmnet:9700']
to_ret={'psi': {'seeds': ['elastic2054.codfw.wmnet:9700', 'elastic2076.codfw.wmnet:9700', 'elastic2080.codfw.wmnet:9700']}}
seeds=['elastic2042.codfw.wmnet:9500', 'elastic2047.codfw.wmnet:9500', 'elastic2052.codfw.wmnet:9500']
to_ret={'omega': {'seeds': ['elastic2042.codfw.wmnet:9500', 'elastic2047.codfw.wmnet:9500', 'elastic2052.codfw.wmnet:9500']}}

EDIT: The above cleared the alerts:

2:29 PM RECOVERY - ElasticSearch setting check - 9400 on elastic2052 is OK: OK - All good! https://wikitech.wikimedia.org/wiki/Search%23Administration
2:29 PM RECOVERY - ElasticSearch setting check - 9200 on elastic2031 is OK: OK - All good! https://wikitech.wikimedia.org/wiki/Search%23Administration
2:29 PM RECOVERY - ElasticSearch setting check - 9200 on elastic2042 is OK: OK - All good! https://wikitech.wikimedia.org/wiki/Search%23Administration
2:29 PM RECOVERY - ElasticSearch setting check - 9200 on elastic2025 is OK: OK - All good! https://wikitech.wikimedia.org/wiki/Search%23Administration
2:29 PM RECOVERY - ElasticSearch setting check - 9400 on elastic2042 is OK: OK - All good! https://wikitech.wikimedia.org/wiki/Search%23Administration
2:29 PM RECOVERY - ElasticSearch setting check - 9400 on elastic2047 is OK: OK - All good! https://wikitech.wikimedia.org/wiki/Search%23Administration
gerritbot added a comment.Sep 29 2022, 6:51 PM

Change 836890 had a related patch set uploaded (by Gehel; author: Gehel):

[operations/puppet@production] elasticsearch: Increase number of master-eligible nodes to 5

https://gerrit.wikimedia.org/r/836890

gerritbot added a comment.Sep 29 2022, 7:00 PM

Change 836908 had a related patch set uploaded (by Gehel; author: Gehel):

[operations/puppet@production] elasticsearch: Increase number of master-eligible nodes to 5 for eqiad

https://gerrit.wikimedia.org/r/836908

gerritbot added a comment.Sep 29 2022, 7:06 PM

Change 836912 had a related patch set uploaded (by Gehel; author: Gehel):

[operations/puppet@production] elasticsearch: Elasticsearch 7 does not need to specify number of masters

https://gerrit.wikimedia.org/r/836912

gerritbot added a comment.Sep 29 2022, 7:08 PM

Change 836890 merged by Gehel:

[operations/puppet@production] elasticsearch: Increase number of master-eligible nodes to 5 for codfw

https://gerrit.wikimedia.org/r/836890

Stashbot added a comment.Sep 29 2022, 7:29 PM

Mentioned in SAL (#wikimedia-operations) [2022-09-29T19:29:02Z] <ryankemper@cumin2002> START - Cookbook sre.hosts.downtime for 3:00:00 on 6 hosts with reason: T313431

Stashbot added a comment.Sep 29 2022, 7:29 PM

Mentioned in SAL (#wikimedia-operations) [2022-09-29T19:29:47Z] <ryankemper@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on 6 hosts with reason: T313431

Stashbot added a comment.Sep 29 2022, 7:55 PM

Mentioned in SAL (#wikimedia-operations) [2022-09-29T19:55:49Z] <ryankemper> T313431 Restarting elasticsearch_7* services on elastic208[1,3] to pick up new master-eligible status

Stashbot added a comment.Sep 29 2022, 8:17 PM

Mentioned in SAL (#wikimedia-operations) [2022-09-29T20:17:25Z] <ryankemper> T313431 Restarting elasticsearch_7* services on elastic2086 to pick up new master-eligible status

Stashbot added a comment.Sep 29 2022, 8:41 PM

Mentioned in SAL (#wikimedia-operations) [2022-09-29T20:41:28Z] <ryankemper> T313431 Restarting elasticsearch_7* services on elastic2080 to pick up new master-eligible status

Stashbot added a comment.Sep 29 2022, 8:58 PM

Mentioned in SAL (#wikimedia-operations) [2022-09-29T20:58:02Z] <ryankemper> T313431 Updated cross-cluster seed conf with new masters; should resolve the settings check alerts

Stashbot added a comment.Sep 29 2022, 8:59 PM

Mentioned in SAL (#wikimedia-operations) [2022-09-29T20:59:21Z] <ryankemper> T313431 Repooled elastic[2073-2074,2080-2081,2083,2086].codfw.wmnet. Codfw's all on 5 masters now and cluster is back to green.

Gehel assigned this task to RKemper.Oct 3 2022, 3:24 PM
Gehel moved this task from Ready for Dev -- SWE to In Progress on the Discovery-Search (Current work) board.
gerritbot added a comment.Oct 5 2022, 5:32 PM

Change 830923 abandoned by Ryan Kemper:

[operations/puppet@production] Revert "elastic: reduce master-eligibles for codfw back down to 2"

Reason:

not needed anymore

https://gerrit.wikimedia.org/r/830923

gerritbot added a comment.Oct 6 2022, 6:42 PM

Change 839668 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper):

[operations/puppet@production] elastic: replace 2 codfw masters to be decom'd

https://gerrit.wikimedia.org/r/839668

gerritbot added a comment.Oct 6 2022, 6:47 PM

Change 839668 merged by Ryan Kemper:

[operations/puppet@production] elastic: replace 2 codfw masters to be decom'd

https://gerrit.wikimedia.org/r/839668

Stashbot added a comment.Oct 6 2022, 6:50 PM

Mentioned in SAL (#wikimedia-operations) [2022-10-06T18:50:14Z] <gehel@cumin2002> START - Cookbook sre.hosts.downtime for 3:00:00 on elastic2061.codfw.wmnet with reason: restarting for config reload - T313431

Stashbot added a comment.Oct 6 2022, 6:50 PM

Mentioned in SAL (#wikimedia-operations) [2022-10-06T18:50:29Z] <gehel@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on elastic2061.codfw.wmnet with reason: restarting for config reload - T313431

Stashbot added a comment.Oct 6 2022, 6:50 PM

Mentioned in SAL (#wikimedia-operations) [2022-10-06T18:50:37Z] <gehel@cumin2002> START - Cookbook sre.hosts.downtime for 3:00:00 on elastic2084.codfw.wmnet with reason: restarting for config reload - T313431

Stashbot added a comment.Oct 6 2022, 6:51 PM

Mentioned in SAL (#wikimedia-operations) [2022-10-06T18:51:04Z] <gehel@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on elastic2084.codfw.wmnet with reason: restarting for config reload - T313431

Stashbot added a comment.Oct 6 2022, 6:52 PM

Mentioned in SAL (#wikimedia-operations) [2022-10-06T18:52:22Z] <gehel@cumin2002> START - Cookbook sre.hosts.downtime for 3:00:00 on elastic[2025,2031].codfw.wmnet with reason: restarting for config reload - T313431

Stashbot added a comment.Oct 6 2022, 6:52 PM

Mentioned in SAL (#wikimedia-operations) [2022-10-06T18:52:39Z] <gehel@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on elastic[2025,2031].codfw.wmnet with reason: restarting for config reload - T313431

Stashbot added a comment.Oct 6 2022, 7:03 PM

Mentioned in SAL (#wikimedia-operations) [2022-10-06T19:03:20Z] <inflatador> 'bking@elastic restarted elastic2025, 2031, 2061, 2084 T313431

gerritbot added a comment.Oct 6 2022, 7:16 PM

Change 839673 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] elastic: raise master-eligibles from 3 to 5

https://gerrit.wikimedia.org/r/839673

gerritbot added a comment.Oct 6 2022, 7:20 PM

Change 839673 abandoned by Bking:

[operations/puppet@production] elastic: raise master-eligibles from 3 to 5

Reason:

https://gerrit.wikimedia.org/r/839673

gerritbot added a comment.Oct 6 2022, 7:23 PM

Change 836908 merged by Bking:

[operations/puppet@production] elasticsearch: Increase number of master-eligible nodes to 5 for eqiad

https://gerrit.wikimedia.org/r/836908

Stashbot added a comment.Oct 6 2022, 7:28 PM

Mentioned in SAL (#wikimedia-operations) [2022-10-06T19:28:53Z] <ryankemper@cumin1001> START - Cookbook sre.hosts.downtime for 3:00:00 on 6 hosts with reason: T313431

Stashbot added a comment.Oct 6 2022, 7:29 PM

Mentioned in SAL (#wikimedia-operations) [2022-10-06T19:29:10Z] <ryankemper@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on 6 hosts with reason: T313431

gerritbot added a comment.Oct 13 2022, 7:26 PM

Change 836912 merged by Ryan Kemper:

[operations/puppet@production] elasticsearch: Elasticsearch 7 does not need to specify number of masters

https://gerrit.wikimedia.org/r/836912

RKemper moved this task from In Progress to Needs Reporting on the Discovery-Search (Current work) board.Oct 13 2022, 7:32 PM
Gehel closed this task as Resolved.Nov 7 2022, 3:44 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits