All versions of Cassandra 3.x tested (3.7.3-instaclustr, 3.10, and 3.11.0 (snapshot)) have all exhibited higher than usual read latency.
Description
Description
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | Eevans | T160570 Cassandra 3.x Tracking | |||
| Duplicate | Eevans | T167477 Unusually high read latency |
Event Timeline
Comment Actions
Discovered so far:
Read throughput was quite high relative to the number of read ops (as compared to production), and this would seem to correlate to an abnormally high number of major page faults.
A warning on startup suggests that the maximum memory maps per process should be increased (significantly), but following the recommendation doesn't seem to help.
WARN [main] 2017-06-08 18:50:39,162 StartupChecks.java:265 - Maximum number of memory map areas per process (vm.max_map_count) 65530 is too low, recommended value: 1048575, you can change it with sysctl.
We had a similar issue after upgrading to 2.2.x (see: T137419: Investigate aberrant disk read throughput in Cassandra (affects 2.2.x and 3.x)), the work-around was to set disk_access_mode: mmap_index_only in cassandra.yaml. While that option has been removed from the default config in 3.11.0, the option itself is still supported, and setting it results in a dramatic difference.
