Slow writes on ixsystems hardware

[Thomas_VDB]

Hi all,
We have bought an ixystems FreeNAS machine (with only hardware support - no software support).

• Neptune 2112 12-bay server
• Xeon Quadcore 3.5Ghz
• 32 GB DDR4 ECC/REG
• 8x SAS 3TB 6Gb/s in RAID10 (12TB usable)
  
• ZIL SSD 128GB Samsung 850 Pro
• L2ARC SSD 256GB Samsung 850 Pro
• OS : FreeNAS 9.10.1
• 2x 10GB NIC

I am going to install 10GBE NIC's in our 2 ESXi servers and use NFS to share the storage with the ESXi hosts.
For the moment I cannot shut down the hosts, so I have mounted the storage via 1 (company network) switch (HP Procurve) - this is a temporary situation.

The problem that I am seeing is this :
- when I perform dd performance test on the freenas via putty, I get incredible results (around 5.4 GB/s - dd if=/dev/zero of=/mnt/tank/ddfile bs=1024k count=20000)
- I have moved 2 Windows VM's from our old SAN to the new FreeNAS SAN. I run 'Parkdale.exe' to estimate disk speed in those VM's :

• Read speed is maxing out on the gigabit network (around 100MB/s)
• Write speed is between 15MB and 25 MB/s

- Also moving a VM (via Vsphere Migrate command) is going at 15MB/s.

Is expected far better write performance with such hardware : the gigabit connection should be the bottleneck.

Can someone please advise?
Thanks,
Thomas.

 

[Thomas_VDB]

Update :
applying zfs set sync=disabled
solves the write performance. However as I have a ZIL SSD, why do I have write performance issues with sync enabled?

 

[Mirfster]

ZIL SSD 128GB Samsung 850 Pro

In all honesty, you do not want to set "sync=disabled". As far as a SLOG goes, did you install that Samsung afterwards?

Got a link to the "Neptune" Server Specs or Info?

P.S. I don't work for nor am affiliated with IXSystems (Just to be clear.. ;))

 

[Mlovelace]

The Samsung 850 is not an appropriate slog device. There are many forum threads on this very subject, so with a little searching you can find a good replacement. Your dd test looks off for the number of drives in your pool. I am guessing that you didn't disable compression on the dataset you ran the dd test on. You must disable compression to get an accurate result for pool write speed.

 

[Thomas_VDB]

Hi,
Thanks for your answers! The Neptune server was suggested and preconfigured by ixsystems with the Samsung SSD's.
This is the link to the neptune server info:
https://www.ixsystems.com/servers/family/?family=Neptune 2100&&server=2112
This is the complete config :

I agree that setting sync=disabled is bad practice. What I am not understanding is that (with sync=standard) we get such bad write performance, while having an SSD as slog. @Mlovelace : Maybe the Samsung 850 is not optimal as an slog, but does this explain this bad performance? Write performance should be better, right? You are right however on the dd-test. I did not disable compression.

 

[Mlovelace]

There are several problems with using the Samsung as a slog; the most obvious being that it is not powerloss protected and can't recover an uncommitted write. Also, the write endurance is not suited for the task and the drive tends to slow down once the internal buffer is exceeded. A more appropriate slog would be the Intel P3700, Intel DC S3710 or even an Intel 750 (poorer write endurance on the 750 than the others). The biggest takeaway here is the lack of powerloss protection is a killer. If the server looses power, hard reboots or kernel panics with an uncommitted write sitting in the slog you can hose your entire pool.

 

[Mirfster]

I would monitor the Disks and see what speeds the SLOG (Samsung 128GB) is indicating during heavy loads.

Agree with @Mlovelace on the Intels. I use the DC S3710 (200GB) models myself.

Also more RAM would be nice. ;)

 

[Mlovelace]

You may be interested in giving the sticky a read...
https://forums.freenas.org/index.php?threads/some-insights-into-slog-zil-with-zfs-on-freenas.13633/

 

[Mlovelace]

I would monitor the Disks and see what speeds the SLOG (Samsung 128GB) is indicating during heavy loads.

Agree with @Mlovelace on the Intels. I use the DC S3710 (200GB) models myself.

I've got the previous gen Intel DC S3700 and it's been rock solid for two years now.

 

[Thomas_VDB]

There are several problems with using the Samsung as a slog; the most obvious being that it is not powerloss protected and can't recover an uncommitted write. Also, the write endurance is not suited for the task and the drive tends to slow down once the internal buffer is exceeded. A more appropriate slog would be the Intel P3700, Intel DC S3710 or even an Intel 750 (poorer write endurance on the 750 than the others). The biggest takeaway here is the lack of powerloss protection is a killer. If the server looses power, hard reboots or kernel panics with an uncommitted write sitting in the slog you can hose your entire pool.

I understand that there is a powerloss safety issue with the samsung SSD. Do you however think my write performance of 15MB/s would go up to > 100MB/s by swapping it with an Intel 750 SSD?

 

[Thomas_VDB]

I would monitor the Disks and see what speeds the SLOG (Samsung 128GB) is indicating during heavy loads.

Agree with @Mlovelace on the Intels. I use the DC S3710 (200GB) models myself.

Also more RAM would be nice. ;)

Even more RAM? I have 32GB !

 

[Mirfster]

Even more RAM? I have 32GB !

FreeNAS Loves RAM.. I have 96GB for one running iSCSI and still want to get that to at least 128. :)

 

[Thomas_VDB]

Hi,
I hate to think that my home freenas (HP microserver) is outperforming this +6k$ system, and that we have to invest some more in expensive SSD's, just to get better write performance than 25MB/s?
(My comparison with my home freenas is a little incorrect because there I am using SMB and not NFS, but you get the idea).

 

[Mirfster]

You should be getting better than what you described as well. That is why I am interested in what the graphs show under workload... Like ARC, Disks, Network, RAM...

 

[Mlovelace]

I understand that there is a powerloss safety issue with the samsung SSD. Do you however think my write performance of 15MB/s would go up to > 100MB/s by swapping it with an Intel 750 SSD?

This is a test I ran with sync=always using the Intel DC S3700 as a slog on a NFS share.

As you can see I am maxing out the write capabilities of the S3700, and being that the 750 is a much faster drive, I would hazard a guess that it would preform better.

Same test with sync=disable on the same NFS share.

 

[Mlovelace]

Do you intend to continue with NFS for VMware datastores? FreeNAS only supports vaai through iSCSI so you won't be getting any vaai storage benefits over NFS mounts.

 

[Thomas_VDB]

Hi,
forst of all, thanks for you advise. I appreciate it!
I am really jealous of your performance :). Hope that I can achieve these numbers as well.

I have now set the sync back to :
zfs set sync=standard
And I am migrating a +200GB VM from the old datastore to the new one.
See below some graphs. I hope that they will bring some answers :

Thomas.

 

[Thomas_VDB]

Migration is going at 7.8MB/s.
during migration I've set sync back to disabled and migration speed went up to 54MB/s.
So the syncing is for sure the bottleneck.

 

[Mirfster]

Guessing that da0 is the SLOG?

 

[Thomas_VDB]

Do you intend to continue with NFS for VMware datastores? FreeNAS only supports vaai through iSCSI so you won't be getting any vaai storage benefits over NFS mounts.

We have been using NFS for a few years now, and were happy with it because it is easy to setup, and because you get to 'see' the files on the SAN. iSCSI seemed like a lot more hastle.
But because iSCSI does not have to sync-issue (and supports VAAI), I will consider it, if we can't get this issue solved. I do hope we can get better result (close to yours).