Change Details

With multiple DCs, we won't have the luxury of read repairs. The nature of WAN networking also means that there is a higher probability of network issues leading to inconsistencies not covered by hinted hand-off. This makes running anti-entropy repairs at regular intervals more important in a multi-DC setting. The issue with repairs is their cost. We need to establish whether 1) there is a chance that traditional full repairs could be cheap enough to schedule them once per reasonable interval (every <n> days, with n <= 7?), or 2) if we can make [incremental repairs](http://www.datastax.com/dev/blog/more-efficient-repairs) work with reasonable repair performance and impact on read latency. The concern here is the conversion of unrepaired sstables to size-tiered compaction, which could increase the number of sstables accessed per read until the next repair merges them into the leveled compaction pool. On the flip side, leveled compaction uses size-tiered compaction in its highest level anyway, and size-tiered does reduce the write load somewhat. ## See also - {T92355}

Hinted hand-off and read-repair are optimizations for converging on consistency, but cannot be considered a 100% solution, making regular anti-entropy repairs a necessity. Establish a mechanism for regular automated invocations of repairs, as well as a documented process detailing when and how to manually execute repairs in the wake of outages. NOTE: [[ https://phabricator.wikimedia.org/T108611 | Previous attempts ]] have made it fairly obvious that full repairs aren't tractable (take too long), so incremental repairs will almost certainly be needed.

With multiple DCs, we won't have the luxury of read repairs.Hinted hand-off and read-repair are optimizations for converging on consistency, The nature of WAN networking also means that there is a higher probability of network issues leading to inconsistencies not covered by hinted hand-off.but cannot be considered a 100% solution, This makes runningmaking regular anti-entropy repairs at regular intervals more important in a multi-DC settinga necessity. The issue with repairs is their cost.Establish a mechanism for regular automated invocations of repairs, We need to establish whetheras well as a documented process detailing when and how to manually execute repairs in the wake of outages. 1) there is a chance that traditional full repairs could be cheap enough to schedule them once per reasonable interval (every <n> days, with n <= 7?), or 2) if we can make [incremental repairs](http://www.datastax.com/dev/blog/more-efficient-repairs) work with reasonable repair performance and impact on read latency. The concern here is the conversion of unrepaired sstables to size-tiered compaction, which could increase the number of sstables accessed per read until the next repair merges them into the leveled compaction pool. On the flip side, leveled compaction uses size-tiered compaction in its highest level anywayNOTE: [[ https://phabricator.wikimedia.org/T108611 | Previous attempts ]] have made it fairly obvious that full repairs aren't tractable (take too long), and size-tiered does reduce the write load somewhat. ## See also - {T92355}so incremental repairs will almost certainly be needed.