Create tools for banning/unbanning Elastic nodes
Closed, ResolvedPublic3 Estimated Story Points
Actions

Assigned To

Authored By

	bking
	Mar 6 2023, 3:23 PM

Description

While working through T322082 , I had to ban and unban Elastic nodes. There is no tooling for this now, so the process is entirely manual, error-prone, and tedious. Creating this ticket to:

Create a cookbook, script, and/or playbook to quickly and safely ban nodes
Create an alert for nodes that remain banned past a certain time threshold

Details

Other Assignee: RKemper

Subject	Repo	Branch	Lines +/-
elasticsearch: Add node ban logic	operations/cookbooks	master	+117 -0
sre.elasticsearch.ban: Improve error message	operations/cookbooks	master	+1 -1
elasticsearch: handle cloudelastic URLs	operations/cookbooks	master	+5 -1
elastic: [WIP] Add node-banning cookbook	operations/cookbooks	master	+70 -0
sre.elasticsearch.rolling-operation: modernize log msgs	operations/cookbooks	master	+3 -3

Customize query in gerrit

Related Objects

Mentioned Here: T335066: Understand "this node stopped indexing” alerts (Elasticsearch)
T322082: Q1:rerack elastic10[53-67]

Event Timeline

bking created this task.Mar 6 2023, 3:23 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 6 2023, 3:23 PM

• MPhamWMF triaged this task as High priority.Mar 6 2023, 4:54 PM

• MPhamWMF moved this task from needs triage to Ops / SRE on the Discovery-Search board.

Change 899789 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper):

[operations/cookbooks@master] sre.elasticsearch.rolling-operation: modernize log msgs

https://gerrit.wikimedia.org/r/899789

gerritbot added a project: Patch-For-Review.Mar 15 2023, 9:39 PM

Change 899789 merged by Ryan Kemper:

[operations/cookbooks@master] sre.elasticsearch.rolling-operation: modernize log msgs

https://gerrit.wikimedia.org/r/899789

API call: curl -s http://localhost:9200/_nodes/stats | jq .nodes[].attributes.row

Maintenance_bot removed a project: Patch-For-Review.Mar 15 2023, 10:10 PM

Gehel moved this task from Ops / SRE to Current work on the Discovery-Search board.Mar 16 2023, 7:29 PM

Gehel edited projects, added Discovery-Search (Current work); removed Discovery-Search.

Change 901298 had a related patch set uploaded (by Bking; author: Bking):

[operations/cookbooks@master] elastic: [WIP] Add node-banning cookbook

https://gerrit.wikimedia.org/r/901298

gerritbot added a project: Patch-For-Review.Mar 20 2023, 8:28 PM

Will probably use this line as inspiration for expanding given input into discrete hosts.

Change 902502 had a related patch set uploaded (by Bking; author: Bking):

[operations/cookbooks@master] elasticsearch: [WIP] Add node ban logic

https://gerrit.wikimedia.org/r/902502

bking set the point value for this task to 3.Mar 27 2023, 3:27 PM

bking moved this task from Incoming to In Progress on the Discovery-Search (Current work) board.

Quickly find the nodes using curl/jq: curl -s https://search.svc.eqiad.wmnet:9243/_nodes/stats | jq '.nodes[] | select(.attributes.row == "B") |.name' | sort

bking claimed this task.Apr 10 2023, 3:18 PM

bking updated Other Assignee, added: RKemper.

Change 901298 abandoned by Ryan Kemper:

[operations/cookbooks@master] elastic: [WIP] Add node-banning cookbook

Reason:

abandoned in favor of https://gerrit.wikimedia.org/r/c/operations/cookbooks/+/902502

https://gerrit.wikimedia.org/r/901298

Change 902502 merged by jenkins-bot:

[operations/cookbooks@master] elasticsearch: Add node ban logic

https://gerrit.wikimedia.org/r/902502

Maintenance_bot removed a project: Patch-For-Review.Apr 18 2023, 8:30 PM

Change 910037 had a related patch set uploaded (by Bking; author: Bking):