Question regarding performance and setup.

[kujinke]

Hi,

I'm new to the ZFS world so bear with me :D.
Last week at work I had some spare time, so ive done some testing with freenas in order to examine performance.
I'm trying to compare performance to some of our in use products (NetApp, Solidfire, VMware VSAN).
Main goal is NFS based Storage to a VMware cluster, Lots of VM's testing, stress testing with FIO over 10's and 100's of vms and mainly alot of random io.


I took out a new HP DL360p Gen9 filled it up with 256GB of RAM and 8 - 2TB enterprise SSD's (Actually 1.92TB - Samsung PM863a), the server also comes with a beast of a dual CPU setup (E5-2687W v3) and a Chelsio T580-LP-CR nic that ive connected to my environment switch.
I set the raid controller to HBA mode and installed FreeNas 11 U3 on a USB Stick.
iperf seems to show pretty solid performance over the network.

I have created a 4x mirrors out of the ssds, and after some google reading I found out that getting a SLOG, and L2 seems as a wasted option on an all flash array.

And to the point - NFS performance to my VMware Cluster had horrible Write speeds, somewhere around 30/40 MB/s while Read speeds kept hitting somewhere around 700-800 MB/s.

Can anyone enlighten me ? Whats wrong with my setup ? and Why write speeds are so bad ?

 

[SweetAndLow]

What's controller are you using? Did you flash it to IT mode? What are your local write speeds? Test using dd.

 

[kujinke]

Controller is: HP P840 and it is set to HBA Mode (IT) all raid functions are disabled and disks are being passed straight to the OS(FreeNAS).
Ive booted up WinPE Based iso and benchmarked the ssd's, performance seems pretty good.. doesnt look like a controller bottleneck or something like that.
a single SSD - 470/450 - read/write - Q32t1 - Crystaldiskmark.

dd on the pool also shows some good results (1.6GB/s) so i dont know why NFS writes are that bad ..

 

[kujinke]

SMB share performance: 700-800mb/s reads and 600mb/s write.

 

[SweetAndLow]

When testing r/w speed you have to use a dataset with compression turned off.

dd if=/dev/zero of=./100g.dat bs=1M count=100000

I'm also curious what version of it firmware you have. What's the output of sas2flash -listall.

 

[kujinke]

root@freenas:~ # dd if=/dev/zero of=/mnt/STOR/STORAGESMB/10g.dat bs=1M count=10000
10000+0 records in
10000+0 records out
10485760000 bytes transferred in 49.706147 secs (210954995 bytes/sec)

 

[kujinke]

Oh sorry that was for a 10gig one.. ill make a 100gig dd.

But now that was strange its around 210MB/sec

 

[SweetAndLow]

Your write speed is pretty slow. It's basically as fast as a single hdd. It's probably your controller that is still in raid mode. You need to be using the mps driver and anything else is wrong. Not only from a performance perspective but a data protection also. You will 100% lose the data if you use a raid0 or jbod mode. You must have a hba in IT mode.

Did you run the sas2flash command I asked for?

 

[nightshade00013]

The P840 doesn't seem to show under supported hardware for FreeBSD 11

https://www.freebsd.org/releases/11.0R/hardware.html#disk

The P830 is along with some others that are close so it is probably half working, but not completely compatible. So basically you will probably have to pull the HP controller and grab something that is 100% compatible.

Maybe look for a 9300-8i or a M1215 which are based on the SAS 3008 or find a SAS 9305-16i which is SAS 3216 based. Since you are using SSD's you probably want to go with something newer so they can reach their full potential vs the SAS 2008 SAS 2308 https://www.servethehome.com/buyers...s-freenas-nas-servers/top-picks-freenas-hbas/ has some great infor for grabbing a SAS controller. And if you grab a M1215 you will want to look at https://www.servethehome.com/flash-lsi-sas-3008-hba-e-g-ibm-m1215-mode/