one ripple effect of multi-instance cassandra will be on graphite metrics, ATM each cassandra JVM uses ~11G on disk in graphite, e.g. restbase1009 (total 222G now)
per-CF metrics take up two-three order of magnitudes more than the rest
```
$ du -hcs restbase1009/org/apache/cassandra/metrics/*
1.5M restbase1009/org/apache/cassandra/metrics/CQL
12M restbase1009/org/apache/cassandra/metrics/Cache
46M restbase1009/org/apache/cassandra/metrics/ClientRequest
9.9G restbase1009/org/apache/cassandra/metrics/ColumnFamily
9.9M restbase1009/org/apache/cassandra/metrics/CommitLog
2.4M restbase1009/org/apache/cassandra/metrics/Compaction
47M restbase1009/org/apache/cassandra/metrics/Connection
14M restbase1009/org/apache/cassandra/metrics/DroppedMessage
3.6M restbase1009/org/apache/cassandra/metrics/FileCache
4.5M restbase1009/org/apache/cassandra/metrics/ReadRepair
1.2M restbase1009/org/apache/cassandra/metrics/Storage
38M restbase1009/org/apache/cassandra/metrics/ThreadPools
10G total
```
and some metric types have plenty of derived metrics which contribute to that, e.g. latencies `CasCommitLatency`
```
-rw-r--r-- 1 _graphite _graphite 309088 Sep 25 13:50 15MinuteRate.wsp
-rw-r--r-- 1 _graphite _graphite 309088 Sep 25 13:49 1MinuteRate.wsp
-rw-r--r-- 1 _graphite _graphite 309088 Sep 25 13:49 50percentile.wsp
-rw-r--r-- 1 _graphite _graphite 309088 Sep 25 13:49 5MinuteRate.wsp
-rw-r--r-- 1 _graphite _graphite 309088 Sep 25 13:51 75percentile.wsp
-rw-r--r-- 1 _graphite _graphite 309088 Sep 25 13:49 95percentile.wsp
-rw-r--r-- 1 _graphite _graphite 309088 Sep 25 13:50 98percentile.wsp
-rw-r--r-- 1 _graphite _graphite 309088 Sep 25 13:50 999percentile.wsp
-rw-r--r-- 1 _graphite _graphite 309088 Sep 25 13:47 99percentile.wsp
-rw-r--r-- 1 _graphite _graphite 309088 Sep 25 13:47 count.wsp
-rw-r--r-- 1 _graphite _graphite 309088 Sep 25 13:47 max.wsp
-rw-r--r-- 1 _graphite _graphite 309088 Sep 25 13:48 mean.wsp
-rw-r--r-- 1 _graphite _graphite 309088 Sep 25 13:49 meanRate.wsp
-rw-r--r-- 1 _graphite _graphite 309088 Sep 25 13:50 min.wsp
-rw-r--r-- 1 _graphite _graphite 309088 Sep 25 13:50 stddev.wsp
```
for each of those there's between 2k and 3k files each of 309kb
```
15MinuteRate.wsp:1389
1MinuteRate.wsp:1389
50percentile.wsp:2141
5MinuteRate.wsp:1389
75percentile.wsp:2141
95percentile.wsp:2141
98percentile.wsp:2141
999percentile.wsp:2009
99percentile.wsp:2141
count.wsp:3548
max.wsp:2141
mean.wsp:2009
meanRate.wsp:1389
min.wsp:2141
stddev.wsp:2009
```
I think we can trim the list of derived metrics to the most relevant ones, e.g. 50/75/95/99 percentile, count, 1MinuteRate