So INTEL Xeon GOLD 6128 3.4GHz has now been fitted to the rig. We re-ran performance tests with DD and real-world. We still just couldn't get 2 x RaidZ2 (8 disks each) to work for the real-world.
I searched around for posts to do with mirrored array speeds and why the DD testing just seemed really odd!I came across this:
https://forums.freebsd.org/threads/zfs-read-performance-of-mirrored-vdevs.59879/
The (above) forum posts are really interesting, for me they seem to indicate that ZFS and mirrors must have some issues. But how that translates into data being processed vs. disk status I'm not too sure about. I still don't really get why writes are quicker than reads though
Then, luckily for me, the client ran real-world testing on the, what I'll now call, a bunch of mirrors (8 2-disk mirrors). Though we still got poor read speeds, it actually seemed to work correctly finally getting some consistent data transfers with good IOPs.
There was a lot of overhead going on with SMB, so ended up using iSCSI instead. We're now, generally speaking, getting around 900Mbytes writes and over 1Gbyte reads per second. And with the IOPs of running mirrors it's chewing through DNG files like they're going out of fashion! I made sure that the record size was set to 1M, and during testing did show a performance increase that was significant.
I've still got a lot to learn ;), but why we never real-world tested the mirrors, I've got no idea. I think the fixation of large transfer speeds always pointed to the RaidZ2 setup, but really we could not get enough IOPs for the DNG files to work correctly with the video editor.
Did we really need the high end CPU? I don't think we'll really find out as the previous processor won't be going back into the rig. So will it FreeNAS? Yes, but pay extra care into IOPs and make sure you do your DD testing
and real-world testing!