Page MenuHomePhabricator

Automated invocation of Cassandra repair jobs
Closed, DuplicatePublic

Description

It is best practice to periodically issue repairs on each node of a Cassandra cluster. These repair jobs result in a great deal of processor utilization, and disk and network IO, and so are not recommended to be run concurrently; Ideally, node repairs should happen sequentially.

One solution would be to install a job that orchestrates sequential repairs from a single node in the cluster. Such a job could be safely invoked from all hosts at a specified time, but only one (selected from the available nodes using a simple heuristic), would actually issue the repairs.

See also

Event Timeline

Eevans created this task.Mar 11 2015, 2:42 AM
Eevans claimed this task.
Eevans raised the priority of this task from to Medium.
Eevans updated the task description. (Show Details)
Eevans added subscribers: Eevans, GWicke, mobrovac.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 11 2015, 2:42 AM
GWicke moved this task from Backlog to In progress on the RESTBase board.Mar 17 2015, 8:16 PM

Change 198297 had a related patch set uploaded (by Eevans):
disable nightly repairs

https://gerrit.wikimedia.org/r/198297

Eevans set Security to None.

Change 198297 merged by Ottomata:
disable nightly repairs

https://gerrit.wikimedia.org/r/198297

Restricted Application added a subscriber: Matanya. · View Herald TranscriptJun 29 2015, 5:26 PM
elukey added a subscriber: elukey.Apr 6 2016, 11:39 AM
Eevans renamed this task from automated invocation of Cassandra repair jobs to Automated invocation of Cassandra repair jobs.Apr 29 2016, 8:29 PM