Prior to moving to production, we should benchmark (and document) the performance of the session storage service under mixed workloads. This information should prove invaluable in validating the implementation, providing a baseline for any subsequent optimizations, and in future capacity planning.
Each machine is a dual Intel Xeon Silver 4110 2.1G (8C, 16T) w/ 64G RAM, 2 @ 128G SSDs, and 1 gbit NIC.
Every node runs a single instance of Cassandra 3.11.2 (6 node cluster). All Cassandra data (commitlog, sstables, etc) shares a RAID-1.
Kask is run from screen session on sessionstore1001, port 8080.
wrk is executed from sessionstore1002. A Lua script is used to create a randomized mixed workload from a pregenerated JSON-formatted file:
$ wrk --latency -t8 -c2048 -d10m -s multi-request-json.lua http://sessionstore1001.eqiad.wmnet:8080 ...
|Threads||Concurrency||Size (k/v)||Ratio (r/w)||Throughput||50p latency||99p latency||Errors|
Caveats, Observations & Comments
- Errors in all cases above were the result of Cassandra timeouts
- No effort has (yet) been made to tune Cassandra for this workload
- Kask and wrk are co-located on two of the Cassandra nodes; Resource contention between the database, application, and benchmark-er influences the results
- As noted in the comments, latency distribution would suggest that something adds ~40ms to ~1/2 of requests (see Figure 1)
|Figure 1: Latency distributions|