This is a simple test that colud have a big impact on the performance of the cluster. It's worth spending some time to check it out.
To test:
- Take one osd out
- Do fio tests on the disk
- Disable write cache
- Do fio tests on the disk
- Reprovision that osd (the disk and process, not the host)
- Add again to the cluster