This is my first post on the forums and I've read countless hours for several weeks, and tried so many settings that I'm finally going to ask for help because I just can't seem to get an expected result from this box.
Hardware setup:
Supermicro X9SRH-7TF w/X540 Intel 10GB NICs
Xeon E5-1650 v2 3.5Ghz Hyperthreaded 6-core CPU
HBA LSI2308 based (using "mps" driver)
128 GB of RAM
There are 18 drives in the machine. 10 SAS 2TB drives, 6 SSDs for a volume, and 2 SSDs for ZIL/SLOG.
2 Different ZFS volumes are configured:
Volume1: 6 SSDs (I tried Raidz2 and Raid1+0)
Volume2: 10 SAS 2TB Drives (I tried Raidz2 and Raid1+0), 2 SSDs mirror'd for ZIL/SLOG.
I ran all of jgreco's tests that he recommended in this thread:
http://forums.freenas.org/index.php?threads/write-performance-issues-mid-level-zfs-setup.13372/
On the FreeNAS shell, running those tests I get amazing results.
For dd writes on the SAS array I get:
1373164797952 bytes transferred in 300.424637 secs (4570746299 bytes/sec)
For dd reads on the SAS array I get:
1373164797952 bytes transferred in 151.147971 secs (9084903935 bytes/sec)
I have lz4 (default) compression on.
Running the iostat against a 262144 block size, each drive is maintaining a fairly steady 148-155MB/s on the SAS volume and 250-255MB/s on the SSD volume. iostat on the SSD volume yields even higher read/write results for the dd test.
Running individual drive tests yields the same results, so there is no bottle neck on the bus or HBA.
iPerf tests between the FreeNAS box and my host yields a steady 9.4-9.6G/bits so there's no issues on the network layer. Physical configuration is FreeNAS box and ESXi host plugged into isolated 10Gig switch, I even tried a 10G cross over cable and connected the host directly to the FreeNAS box and eliminated the switch from the equation. No change in the results.
I've run autotune off and autotune on, I've tried net.inet.tcp.delayed_ack=0 and kern.ipc.nmbclusters at various levels, no change in my results.
Autotune set these syctls for me:
kern.ipc.maxsockbuf = 2097152
net.inet.tcp.recvbuf_max = 2097152
net.inet.tcp.sendbuf_max = 2097152
Tunables set by Autotune:
vfs.zfs.arc_max = 97174518588
vm.kmem_size = 107971687321
vm.kmem_size_max = 134964609152
I've tried NFS and iSCSI based reads/writes and when I try to do dd based tests with the FreeNAS box mounted to the host I average 500-515M/Byte writes (doing 1M and 4M block sizes) and 330-340M/Byte reads. (Which makes no sense, reads should be faster...)
I've tried with and without the SLOG, and see no difference in performance. Memory usage is at a fairly steady 30Gb. CPU has never gone above 20%.
In iSCSI mode, with sync=standard (the default) I see these results above. If I set sync=always my write speeds drop to 31MByte/s. My NFS speeds are similar in default mode (even with the SLOG), and if I disable sync it performs similar to iSCSI in normal mode.
I'm truly at a loss here, on the FreeNAS box, on the command line the drives are performing at incredible rates, but when connecting to the box via NFS/iSCSI performance is 1/10th to 1/25th of the on-box levels.
So my question is... how bad is the ESXi overhead, and is there any additional settings/tweaks I can make or tests I can run to try and isolate the issue?
Should the variance of on-box vs off-box performance be this extreme? I've had standard hardware RAID setups perform way better than this, so I feel that something just isn't right.
Any assistance is appreciated.
Thanks much!
-F
Hardware setup:
Supermicro X9SRH-7TF w/X540 Intel 10GB NICs
Xeon E5-1650 v2 3.5Ghz Hyperthreaded 6-core CPU
HBA LSI2308 based (using "mps" driver)
128 GB of RAM
There are 18 drives in the machine. 10 SAS 2TB drives, 6 SSDs for a volume, and 2 SSDs for ZIL/SLOG.
2 Different ZFS volumes are configured:
Volume1: 6 SSDs (I tried Raidz2 and Raid1+0)
Volume2: 10 SAS 2TB Drives (I tried Raidz2 and Raid1+0), 2 SSDs mirror'd for ZIL/SLOG.
I ran all of jgreco's tests that he recommended in this thread:
http://forums.freenas.org/index.php?threads/write-performance-issues-mid-level-zfs-setup.13372/
On the FreeNAS shell, running those tests I get amazing results.
For dd writes on the SAS array I get:
1373164797952 bytes transferred in 300.424637 secs (4570746299 bytes/sec)
For dd reads on the SAS array I get:
1373164797952 bytes transferred in 151.147971 secs (9084903935 bytes/sec)
I have lz4 (default) compression on.
Running the iostat against a 262144 block size, each drive is maintaining a fairly steady 148-155MB/s on the SAS volume and 250-255MB/s on the SSD volume. iostat on the SSD volume yields even higher read/write results for the dd test.
Running individual drive tests yields the same results, so there is no bottle neck on the bus or HBA.
iPerf tests between the FreeNAS box and my host yields a steady 9.4-9.6G/bits so there's no issues on the network layer. Physical configuration is FreeNAS box and ESXi host plugged into isolated 10Gig switch, I even tried a 10G cross over cable and connected the host directly to the FreeNAS box and eliminated the switch from the equation. No change in the results.
I've run autotune off and autotune on, I've tried net.inet.tcp.delayed_ack=0 and kern.ipc.nmbclusters at various levels, no change in my results.
Autotune set these syctls for me:
kern.ipc.maxsockbuf = 2097152
net.inet.tcp.recvbuf_max = 2097152
net.inet.tcp.sendbuf_max = 2097152
Tunables set by Autotune:
vfs.zfs.arc_max = 97174518588
vm.kmem_size = 107971687321
vm.kmem_size_max = 134964609152
I've tried NFS and iSCSI based reads/writes and when I try to do dd based tests with the FreeNAS box mounted to the host I average 500-515M/Byte writes (doing 1M and 4M block sizes) and 330-340M/Byte reads. (Which makes no sense, reads should be faster...)
I've tried with and without the SLOG, and see no difference in performance. Memory usage is at a fairly steady 30Gb. CPU has never gone above 20%.
In iSCSI mode, with sync=standard (the default) I see these results above. If I set sync=always my write speeds drop to 31MByte/s. My NFS speeds are similar in default mode (even with the SLOG), and if I disable sync it performs similar to iSCSI in normal mode.
I'm truly at a loss here, on the FreeNAS box, on the command line the drives are performing at incredible rates, but when connecting to the box via NFS/iSCSI performance is 1/10th to 1/25th of the on-box levels.
So my question is... how bad is the ESXi overhead, and is there any additional settings/tweaks I can make or tests I can run to try and isolate the issue?
Should the variance of on-box vs off-box performance be this extreme? I've had standard hardware RAID setups perform way better than this, so I feel that something just isn't right.
Any assistance is appreciated.
Thanks much!
-F