Only getting about 55MB/s writes on RAIDZ2 NFS with 6x 7200RPM disks.. is that normal?

[Beer]

I'm kind of disappointed at how my new build is performing. I had done a lot of research in advance about performance and I'm getting 1/3rd to 1/4 of the performance I was expecting out of this.

Here is my setup:

Xeon E5-2630L V3
32GB DDR4 ECC (24GB dedicated to FreeNAS)
ESXi 6.5 on USB
FreeNAS 9.10 (virtualized) on 2x USB datastores
2x 256GB SSD @onboard sata
6x 4TB HDD @onboard sata
Entire onboard SATA controller passed through to FreeNAS
2x SSDs are mirrored and shared back over to ESXi to host other VMs, irrelevant
6x HDDs in RAIDZ2
NFS shares for media/backups on HDDs

Here's a pic of my config

I've tried enabling autotune, but that seemed to have no effect whatsoever.

What else can I do to squeeze some better performance out of this?

 

[Spearfoot]

I'm kind of disappointed at how my new build is performing. I had done a lot of research in advance about performance and I'm getting 1/3rd to 1/4 of the performance I was expecting out of this.

Here is my setup:

Xeon E5-2630L V3
32GB DDR4 ECC (24GB dedicated to FreeNAS)
ESXi 6.5 on USB
FreeNAS 9.10 (virtualized) on 2x USB datastores
2x 256GB SSD @onboard sata
6x 4TB HDD @onboard sata
Entire onboard SATA controller passed through to FreeNAS
2x SSDs are mirrored and shared back over to ESXi to host other VMs, irrelevant
6x HDDs in RAIDZ2
NFS shares for media/backups on HDDs

Here's a pic of my config

I've tried enabling autotune, but that seemed to have no effect whatsoever.

What else can I do to squeeze some better performance out of this?

Interesting... I wonder if your problem has something to do with passing through the motherboard SATA controller to the FreeNAS VM? Most All-in-One builders use a HBA, though passing through the motherboard SATA subsystem is supposed to work... in theory. What brand of motherboard are you using?

 

[Beer]

Interesting... I wonder if your problem has something to do with passing through the motherboard SATA controller to the FreeNAS VM? Most All-in-One builders use a HBA, though passing through the motherboard SATA subsystem is supposed to work... in theory. What brand of motherboard are you using?

It's an ASRock Rack EPC612D4U
http://www.asrockrack.com/general/productdetail.asp?Model=EPC612D4U#Specifications

Onboard sata uses Intell Wellsburg AHCI Controller. My mirrored SSDs perform just fine.. so I was looking at my RAIDZ2 setup being the issue

 

[bigphil]

It depends on how you're accessing the NFS share. Are you mounting it as a datastore in ESXi or is it being mounted by a different client? The problem is most likely caused by sync writes to the spinning disks. The SSD's aren't seeing it because of the obvious speed vs rust. On the dataset having the slow performance, temporarily set the zfs property "sync" to be sync=disabled (zfs set sync=disabled yourpool/yourdataset). If doing this improves the performance, the preferred method to mitigate this is to get an SSD or NVMe drive that has power loss protection and configure it as a SLOG device for that pool and then set the sync property back to standard or always (depending on the use case).

 

[Beer]

It's going to be a while before I can get back to you on this but I will definitely try that and reply back. Midnight right now and work in the AM, and I will be out of town until Saturday. Thanks a ton for the suggestion

 

[Spearfoot]

Ha! Just realized you're running an All-in-One! Dooooh!

The thing to do for NFS or iSCSI VM storage in an AIO is to set up a second vSwitch in ESXi dedicated for storage use. @Benjamin Bryan's 'How-To' article (FreeNAS 9.10 on VMware ESXi 6.0) goes into great detail about setting this up. Briefly, you use the dedicated vSwitch to establish a really fast connection between ESXi and your FreeNAS-based NFS or iSCSI datastores, separate from your VM Network. Here's a screenshot of my setup, showing how it works. I get blazingly fast disk I/O on my VMs using this technique.



EDIT: Here are the CrystalDiskMark results of a VM running on an NFS-based datastore using the vSwitch method described above. Interface vmx1 is the storage network; see the Tx spiking over 8Gb/s? Writes from the VM are significantly slower, because I've turned synchronous writes 'on' for the dataset and am using a SLOG device. Not too bad for spinning rust!

 

[SweetAndLow]

What workflow? What does iperf give you? What are your local speeds using dd?

 

[Beer]

Maybe my issue is somewhere else.

iperf shows a consistent 112MB/s, dd shows ~111MB/s. Both seem to be saturating my network. That seems good

Code:

$ time dd if=/dev/zero of=/media/nas/testfile bs=16k count=1048576  
1048576+0 records in 
1048576+0 records out 
17179869184 bytes (17 GB, 16 GiB) copied, 155.218 s, 111 MB/s 

real  2m35.222s 
user  0m0.080s 
sys  0m5.080s

Rsync reports 76.83MB/s

Code:

$ rsync --progress testfile /media/nas 
testfile 
17,179,869,184 100%  76.83MB/s  0:03:33 (xfr#1, to-chk=0/1)

I use KDE on my main desktop.. and when I drag files over to my NAS from within Dolphin file browser, the speeds always show around 50-55MB/s. So I guess I need to look elsewhere.