...so read repairs can offer a small improvement over the status quo of no repairs at all.

How are you quantifying this? How small is small?

In T153123#2870783, @Eevans wrote:

...so read repairs can offer a small improvement over the status quo of no repairs at all.

How are you quantifying this? How small is small?

As far as I know, we have metrics for the read repair rate. Do you mind checking those?

In T153123#2871030, @GWicke wrote:

In T153123#2870783, @Eevans wrote:

...so read repairs can offer a small improvement over the status quo of no repairs at all.

How are you quantifying this? How small is small?

As far as I know, we have metrics for the read repair rate. Do you mind checking those?

I think you may have misunderstood. I meant, for any given value(s) where is the cutoff for small (or large, or whatever). How are you quantifying "small"? I'm not saying you're wrong to categorize it as such, I'm just trying to understand.

I personally rate the benefit as "small" considering the limited coverage, but reasonable people can disagree on such ratings. I don't think anybody would consider read repairs to be as reliable and complete as regular full (or incremental) repairs. But at the same time, until we have those, there is clearly a non-zero benefit in read repairs.

Does this outweigh other potential benefits of disabling read repairs like less manual time spent on maintaining this, better performance, or lower disk usage? I personally doubt it, but this task is the place to debate this.

OK, so to be clear. It is technically not possible to disable read repair; Read repair will run whenever digest mismatches occur, and there isn't any knob to turn this off. The read_repair_chance parameter establishes a probabilistic check of the digests of replicas not involved in achieving your consistency level.

You might be able to implicitly disable RR if you set read_repair_chance to zero, and only ever queried at CL = ONE. This would probably run afoul of speculative retries though, and require that it be disabled as well (more investigation would be needed).

In production, we query at localQuorum (except secondary indexes updates which use localOne).

In T153123#2872867, @Eevans wrote:

OK, so to be clear. It is technically not possible to disable read repair; Read repair will run whenever digest mismatches occur, and there isn't any knob to turn this off. The read_repair_chance parameter establishes a probabilistic check of the digests of replicas not involved in achieving your consistency level.

You might be able to implicitly disable RR if you set read_repair_chance to zero, and only ever queried at CL = ONE. This would probably run afoul of speculative retries though, and require that it be disabled as well (more investigation would be needed).

In production, we query at localQuorum (except secondary indexes updates which use localOne).

To add a little more information to this:

The blocking metric counts all read repairs that occur because of a digest mismatch, for us that is about 2/s per node.

The background metric are read repairs that were initiated by chance (read_repair_chance).

Changing the consistency level is not an option in general.

Since Cassandra does not currently allow disabling read repairs, it appears that the best we can do to stop grossly out-of-order data being mixed into SSTables on an ongoing basis would be to get full repairs working.

In any case, it probably would not hurt to set read_repair_chance to zero, and see if it does make any difference to out-of-order writes. It does not seem likely, but it's easy enough to verify.

@Eevans: Could you determine whether there is something we can do about repairs, and then either close this task, or clearly spell out the actionable part?