Maniphest T219799

Create cookbook to reset readonly indices on elasticsearch clusters
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Gehel
	Apr 1 2019, 3:48 PM

Description

As seen in T219364, indices can move to readonly state when disk space is low. This needs to be reset manually once disk space is available again. This should be a cookbook.

Details

Related Changes in Gerrit:

	Subject	Repo	Branch	Lines +/-
	elasticsearch: reset all indices to read/write	operations/cookbooks	master	+23 -0
	elasticsearch: reset all indices to read/write	operations/software/spicerack	master	+44 -0

Customize query in gerrit

Related Objects

Mentioned In: rCCKB58539519f777: elasticsearch: reset all indices to read/write
Mentioned Here: T219364: Elasticsearch indices went read-only causing huge lag

Event Timeline

Gehel created this task.Apr 1 2019, 3:48 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 1 2019, 3:48 PM

greg moved this task from Active investigation to Follow-up prevention on the Wikimedia-Incident board.Apr 2 2019, 6:24 PM

Change 502218 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/software/spicerack@master] elasticsearch: reset all indices to read/write

https://gerrit.wikimedia.org/r/502218

gerritbot added a project: Patch-For-Review.Apr 8 2019, 1:31 PM

Change 502220 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/cookbooks@master] elasticsearch: reset all indices to read/write

https://gerrit.wikimedia.org/r/502220

Gehel moved this task from Incoming to Needs review on the Discovery-Search (Current work) board.Apr 8 2019, 5:39 PM

fgiunchedi triaged this task as Medium priority.Apr 9 2019, 8:37 AM

Gehel claimed this task.Apr 9 2019, 5:17 PM

Change 502218 merged by Gehel:
[operations/software/spicerack@master] elasticsearch: reset all indices to read/write

https://gerrit.wikimedia.org/r/502218

Gehel moved this task from Needs review to Needs Reporting on the Discovery-Search (Current work) board.Apr 11 2019, 12:13 PM

• debt closed this task as Resolved.Apr 15 2019, 5:37 PM

Change 502220 merged by Gehel:
[operations/cookbooks@master] elasticsearch: reset all indices to read/write

https://gerrit.wikimedia.org/r/502220

Krinkle edited projects, added Sustainability (Incident Followup); removed Wikimedia-Incident.Apr 28 2020, 9:50 PM

Maintenance_bot removed a project: Patch-For-Review.Apr 28 2020, 10:16 PM

Gehel mentioned this in rCCKB58539519f777: elasticsearch: reset all indices to read/write.Dec 14 2022, 3:24 PM

Create cookbook to reset readonly indices on elasticsearch clustersClosed, ResolvedPublicActions

Description

Details

Related Objects

Event Timeline

Create cookbook to reset readonly indices on elasticsearch clusters
Closed, ResolvedPublic
Actions