Cassandra leverages a number of mechanisms in order to converge on a consistent state (read-repair, hinted-handoff, etc). However, it is still necessary to periodically issue repairs to keep any missed inconsistencies from accumulating.
Given that repairs have never really been performed on this cluster, and given the known history of (possibly widespread) [[https://phabricator.wikimedia.org/T102015#1442528|inconsistent deletes]], we can expect the initial repair to be quite extensive. The advantage of starting a repair sooner, rather than later, is that the current state of the three new nodes (restbase100[7-9].eqiad) are likely quite close to what they should be (having just recently concluded bootstrapping). Additionally, the overall size of the cluster dataset is currently low, and can only be expected to get larger.
Prior to enabling [[https://phabricator.wikimedia.org/T92355|automated, recurring anti-entropy repairs]] (T92355). We should begin with one or more closely monitored manual invocations.
Related: T92355
----
| node | status | started | completed | comments |
|------|------|------|------|------|
|restbase1001|complete|2015-09-09T13:05:39+0000| 2015-09-12T06:56:37+0000| |
|restbase1002|in-progress|2015-09-12T19:22:11+0000| |rack A |
|restbase1003|in-progress |2015-09-14T12:51:25+0000| |rack B |
|restbase1004| | | | |
|restbase1005| | | | |
|restbase1006| | | | |
|restbase1007| | | | |
|restbase1008| | | | |
|restbase1009| | | | |