Do a general look around and see how can we improved the current performance
Some additional context
To add some specific watermarks that would be good to keep in mind around the place we are seeing the most pain around ceph performance (etcd), besides the etcd issues themselves (which can also be influenced by the VMs and the network connection between them), we also should generally treat it as a fairly heavily burdened cluster due to the number of namespaces, number of etcd nodes (more nodes = more demanding) and the amount of traffic on the API. From https://etcd.io/docs/v3.4.0/op-guide/hardware/ we see that a healthy cluster *should* need just 50 sequential IOPS if it doesn't do much. We'd be a lot happier with closer to "For heavily loaded clusters, 500 sequential IOPS (e.g., a typical local SSD or a high performance virtualized block device) is recommended.".
Note: our issues often happen on toolsbeta where the cluster is only conducting minimal automated tasks, so IOPS will help, but latency, iowait and throughput may be bigger issues.
General objectives to aim toward:
- 90th percentile of fsync/fdatasync is less than 10ms more or less consistently
- higher sequential IOPS than we are seeing (how close can we get to 500 or at least for our throttles to matter?). In initial testing ceph was faster than local disk without throttles and now it is nowhere near it. We may have changed VM processor settings and things since then to help enable live migration (worth checking there if raw ceph performance just doesn't line up with VM disk performance).
- iowait is not constantly at whole numbers on etcd nodes
Test used to check the performance:
fio --name=fio-seqrw --bs=64k --direct=1 --filename=/tmp/fio.seqrw --fsync=256 --gtod_reduce=1 --iodepth=64 --ioengine=libaio --rw=rw --size=5G --group_reporting
for fsync:
fio --rw=write --ioengine=sync --fdatasync=1 --directory=/tmp/test-data --size=22m --bs=2300 --name=mytest
Storage performance test results
- etcd-4 vs etcd-8: