Looking at the first production data it looks like our current RESTBase Cassandra cluster does not have a lot of margin on IO bandwidth and storage capacity. With more tuning we might be able to stretch it a bit more, but it is clear that we'll need more capacity sooner rather than later.
We are leaning towards moving to a setup with multiple Cassandra instances per hardware node, aiming for a maximum compressed load of ~600G per instance. With 1T SSDs this could be configured as:
- one instance per physical SSD, JBOD
- all instances sharing a RAID-0 (current setup)
- some other RAID level
JBOD would give us a good amount of failure isolation, but also does not allow us to share the aggregate IO bandwidth between instances. RAID levels other than RAID-0 lose disk capacity, and are not strictly needed as the data is already replicated 3-way across the cluster.
For the current cluster, the main decision we have to make is whether we want to add one SSD to the existing nodes or not. (There is one slot left.)
The exact amount of IO capacity and storage space needed depends on the results of the work outlined in T93751, but I think it is relatively clear that we need more nodes. The next increment that makes sense from a replica placement perspective would be to add three nodes.