This work did not get persisted to the task here so I will attempt a brief outline for posterity. The main difficulty here is we could not take labstore1001 offline to get baseline readings on performance. That makes any comparison apples to oranges in relation to the new setup. We used fio and dd to run through a lot of test scenarios: read, read/write, write for local and from NFS clients end-to-end. The conclusion was the old setup is a tiny bit faster in some cases like large metadata operations (which is currently sucky because with lookupcache disabled which we hope to change soon), but the performance between the two was not obviously a blocker for migration. We have been entirely on the new cluster for Tools since 14/11/2016 and have not encountered notable performance issues. Our TC setup was extended to include the new cluster here.

The biggest win is on the NFSd server itself where we are not running at high load to service normal operations. It is possible to run find and local operations without entirely tanking the setup. So that's nice.

Tuning this will be a long-term consideration. I am going to resolve this task but we'll know more over the coming weeks.

https://linux.die.net/man/1/fio
https://github.com/chasemp/fiorun