NFS datastore in ESX 6.5 from Freenas performance

cvega666

Cadet
Joined
Oct 15, 2016
Messages
4
Hi Folks,

I've put together a new machine that I will use for ESXi. It runs a Z68 motherboard, i7-3770, 20GB ram and a IT flashed HP 220 card - which I am passing through to the Freenas VM.
I have 3 x 1TB WD Purple drives that are in raid5. That volume is shared via NFS - which is then used as a NFS datastore on ESXi. Making sense so far I hope.

This is where issues begin. When i create a VM and use that datastore to host it, the performance inside the VM is .. slow. 30mb/s roughly.
On the other hand, when I access the same NFS share over the network, I get about 100mb/s. I'd expect the other way around..
it's as if there's a bottleneck in ESX when it comes to that share, or something not set up correctly.
 

ian351c

Patron
Joined
Oct 20, 2011
Messages
219
You will definitely want to google that since it's a well trodden path. Basically, all NFS access from VMWare is done in a way to ensure that data is written to disk, which means flushing all cache on every write. This makes things much slower, especially with a setup like yours. You should find plenty of commentary on VMWare over NFS on FreeNAS, discussing possible solutions with various levels of performance and safety for your VMs.
 

cvega666

Cadet
Joined
Oct 15, 2016
Messages
4
Thanks, I had a suspicion this was a ESX thing. I'll do some googling - I'm not a fan of disabling sync as it kinda negates the purpose of the array.
 

HoneyBadger

actually does care
Administrator
Moderator
iXsystems
Joined
Feb 6, 2014
Messages
5,110
The short answer is "you'll need an SLOG device to handle the sync write workload." There's a few different options but it isn't as simple as "attach any random consumer SSD" - check the link in my signature for more information.

Longer answer would involve getting away from RAIDZ and onto mirrors as well since that would provide better steady-state performance for random I/O which virtualization tends to generate.
 

Heracles

Wizard
Joined
Feb 2, 2018
Messages
1,401
Hi,

I would also say that you would probably do way better with iSCSI instead of NFS and Raid10 instead of Raid5. Add these buzzwords to your googling and you should find even more.

Have fun with your setup,
 

John Doe

Guru
Joined
Aug 16, 2011
Messages
635
I am running a similar setup. 2 mirrored SSDs are passing nfs storage back to esxi. runs very well without witchcraft slogs
 

rvassar

Guru
Joined
May 2, 2018
Messages
971
NFS requires, per the RFC, O_SYNC writes. It's been that way since NFS v2 and was the reason for the Sun Microsystems Prestoserve SBUS card, back in the early 90's...

You're better off running a data store on iSCSI. It's fairly easy to set up, and performs much better.
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
Sun Microsystems Prestoserve SBUS card, back in the early 90's...

Ooo! Ooo! PrestoServe *and* Online DiskSuite! I think I may still have a Presto card laying around here somewhere, probably in some poor IPC or other unlucky Sun4...
 

rvassar

Guru
Joined
May 2, 2018
Messages
971
I think I may still have a Presto card laying around here somewhere, probably in some poor IPC or other unlucky Sun4...

The battery is no doubt, dead... ;)
 

Yorick

Wizard
Joined
Nov 4, 2018
Messages
1,912
You're better off running a data store on iSCSI. It's fairly easy to set up, and performs much better.

I’ve seen comments that for block storage, ZFS shouldn’t be filled past 50%, and ideally stay within 10-15%; whereas for file storage, one shouldn’t go beyond 80%.

Keeping at least 50% free is a big chunk. Depending on the amount of data in that data store, maybe NFS plus SLOG works better, then.

What’s your experience with block storage and ZFS “filling up”?
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
I’ve seen comments that for block storage, ZFS shouldn’t be filled past 50%, and ideally stay within 10-15%; whereas for file storage, one shouldn’t go beyond 80%.

Keeping at least 50% free is a big chunk. Depending on the amount of data in that data store, maybe NFS plus SLOG works better, then.

What’s your experience with block storage and ZFS “filling up”?

It's not "filling up." It "slows down."

https://www.ixsystems.com/community/threads/zfs-fragmentation-issues.11818/
 

Yorick

Wizard
Joined
Nov 4, 2018
Messages
1,912
"Slowing down as it fills up" was what I meant :). If you like, "slowing down as occupancy rate rises", better?

Thanks! Post #9 in your link is exactly what I was looking for. The feedback on all-flash performance is also fascinating. It sounds like once moving heads are out of the picture, an occupancy rate of 80 or even 90 percent is fine again.

Followup question if I may. What's so compelling about iSCSI that folk are happy to meet the requirements for it, over NFS?

A few I can think of:
- We've standardized on it
- Multipathing is very nice
- My app is better supported / only supports iSCSI
- My app performs much better on iSCSI
- We've always done things this way
 
Last edited:

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
"Slowing down as it fills up" was what I meant :). If you like, "slowing down as occupancy rate rises", better?

If we want to be pedantic, it is actually slowing down as fragmentation increases, in combination with the occupancy rate.

The part that can fool you is that on a fresh pool, you can really run all the way out to 95%+ at full speed, and if you never wrote anything to it again, reads would also be at full speed because there'd be no fragmentation. People benchmark on fresh pools, which is, just to be blunt, kinda pointless and/or stupid, because that will give you highly sunshine-fulled results that are not representative.

It's the small block rewrites that get you. And I've talked about this endlessly as well. You'll often see me refer to a Delphix graph. This graph shows "steady state" behaviour, which is where you run a pool at a given percent full. The steady state is "as bad as it gets" (or at least is as bad as it is likely to get).

Because this forumware is retarded, it won't let me insert the image anymore.

https://extranet.www.sol.net/files/freenas/fragmentation/delphix-small.png

So it isn't saying that at 50% or 90% your pool will immediately suck. It's saying that over time, as lots of writes and frees happen, your performance will be much better if you're at 25% capacity than 90% capacity. The horrifying thing is that there isn't a ton of difference between 50 and 90%. But also, if you look at 10%, you get AMAZING write speeds, even for random data, because ZFS is basically transforming most writes into sequential writes.

Thanks! Post #9 in your link is exactly what I was looking for.

Followup question if I may. What's so compelling about iSCSI that folk are happy to meet the requirements for it, over NFS?

A few I can think of:
- We've standardized on it
- Multipathing is very nice
- My app is better supported / only supports iSCSI
- My app performs much better on iSCSI
- We've always done things this way

You forgot the common one: :smile:

For block storage such as VM's or databases, the problem is similar for both NFS and iSCSI, as are the requirements.

https://www.ixsystems.com/community...res-more-resources-for-the-same-result.28178/

The thing is, the idea that NFS and iSCSI work out to requiring similar resources is not obvious to some people but obvious to others. My article here is really trying to say that large blobs of storage that involve random writes to portions in the middle, etc., present a very different challenge than traditional NAS-style "store these files" usage. ZFS gets to be aware of the context when you create a media library on your NAS and store large files. Storing the file is done efficiently and as contiguously as free space allows. The files are static and you won't be frequently updating random little blocks inside the files. Minimal impact on fragmentation.

This should never, ever be confused with storing VMDK or database files. This is challenging regardless of NFS or iSCSI. Block rewrites increase frag. It is THIS that actually requires more resources. You need more ARC to hold more metadata, because free space is much more scattershot across the pool. You need more ARC and L2ARC to mitigate poor read speeds due to increased fragmentation. Etc.
 

Mlovelace

Guru
Joined
Aug 19, 2014
Messages
1,111
Followup question if I may. What's so compelling about iSCSI that folk are happy to meet the requirements for it, over NFS?
iSCSI is compelling, where freeNAS is concerned regarding ESXi, because it supports VAAI primitives whereas NFS does not. Some vendors support VAAI with NFS via a plugin, but freeNAS has stated that they are not interested in creating one.

Edit: I suppose it's also worth mentioning that VAAI is only supported on ESXi if the host is licensed for it. Meaning that if you are running the free version of the hypervisor then VAAI is not available.
 

rvassar

Guru
Joined
May 2, 2018
Messages
971
iSCSI is compelling, where freeNAS is concerned regarding ESXi, because it supports VAAI primitives whereas NFS does not. Some vendors support VAAI with NFS via a plugin, but freeNAS has stated that they are not interested in creating one.

Edit: I suppose it's also worth mentioning that VAAI is only supported on ESXi if the host is licensed for it. Meaning that if you are running the free version of the hypervisor then VAAI is not available.

I would consider iSCSI compelling vs NFS for more reasons than this. Start with the CHAP security provided by iSCSI, yes it could be better. But consider, NFS provides essentially "No File Security". You're expected to separate your NFS traffic.

Even when VAAI primatives are not available, I believe all iSCSI devices support:
- Atomic test/set
- Full Copy
- Block Zero
- Thin Provision
- Unmap

Then you have to compare the filesystems. ESXi is going to create a VMDK5 filesystem on either raw zvol storage, or one with the added layer indirection of a plain file on a POSIX filesystem, complete with all the atime/mtime metadata stuff happening needlessly under the covers. Then you arrive at the VMDK layer itself, and the optimizations it may or may not perform based on the state information the Hypervisor has on the individual operation. I may be mistaken, but I believe NFS strips this last bit away via RFC compliance pedantry.
 

Mlovelace

Guru
Joined
Aug 19, 2014
Messages
1,111
You're expected to separate your NFS traffic.
You should absolutely separate all storage traffic iSCSI or NFS, minimally by vLAN, but ideally separate switches.
Even when VAAI primatives are not available, I believe all iSCSI devices support:
- Atomic test/set
- Full Copy
- Block Zero
- Thin Provision
- Unmap
If VAAI primitives are not supported on the target (which not all iSCSI targets support), how are the VAAIs you listed above suppose to be executed? This is a contradictory statement. ;) It would be nice if all iSCSI target implementations supported the VMware APIs, but sadly they do not.
CHAP security
Haha sorry this was kind of funny, but seriously, you could use the kerberos authentication in NFS v4 if you think CHAP is even worth configuring.

create a VMDK5 filesystem
I'm assuming you meant VMFS not VMDK, and there are pros and cons to both VMware datastores and native file access on the storage side; which is why VVOLs are amazing. Though I don't expect freeNAS to develop VVOL support... maybe trueNAS will some day.
 

rvassar

Guru
Joined
May 2, 2018
Messages
971
If VAAI primitives are not supported on the target (which not all iSCSI targets support), how are the VAAIs you listed above suppose to be executed? This is a contradictory statement. ;) It would be nice if all iSCSI target implementations supported the VMware APIs, but sadly they do not.

I did not call those VAAI primatives. ;)

iSCSI == IP enabled Small Computer System Interface. It implements the command set of the old physical SCSI interface over IP. There's something like 60+ commands. I haven't worked in this space in half a decade so a bunch of it has been garbage collected, but... I know unmap was added to support flash devices. My point was to compare with what's possible over NFS. With NFS you get POSIX file handle operations, open/creat/close/lseek/read/write 100% O_SYNC, vs. iSCSI implementing the full 60+ SCSI commands... That's before you start adding VAAI's...

Haha sorry this was kind of funny, but seriously, you could use the kerberos authentication in NFS v4 if you think CHAP is even worth configuring.

iSCSI can be Kerberized as well, though I can't say I have any experience with it. But at least some kind of authentication is there in the original RFC. NFS has always been kind of the wild west... (And this requires a certain amount of Corvidae Cuisine... I'm ex-Sun)

I'm assuming you meant VMFS not VMDK,

Yes.
 

HoneyBadger

actually does care
Administrator
Moderator
iXsystems
Joined
Feb 6, 2014
Messages
5,110
I'll see if I have some time to poke around and get numbers, but the advent of (relatively) inexpensive 10GbE has made the argument fairly moot for a lot of clients. As long as you have proper (non-blocking with flow control) switches in the middle, it's generally "margin of error" between the two if you're using software initiators. Hardware-assisted iSCSI will beat it out in terms of lower initiator CPU usage, but that relies on the initiator's OS behaving properly and not having some funky bugs in the offload that cause certain cards to just choke and die. glares menacingly at vSphere 5.x

From a ZFS perspective, if you build NFS on a dataset with a smaller recordsize then you don't have the partial-record-rewrite issue hammering you on thick disks, and the ability to do some very easy snapshotting. Reclaiming space from a deleted VM is also easy since from the ZFS perspective it's just "delete a file." iSCSI has MPIO in its corner, as well as the in-guest UNMAP that's in newer ESXs. Auto-space-reclaim in VMFS6 is also a boon.

But I don't know why everyone's arguing about NFS vs iSCSI when they really just need to accept that FC will always be superior. ;)

/throws kerosene on the storage-war fire and runs away laughing
 

rvassar

Guru
Joined
May 2, 2018
Messages
971
But I don't know why everyone's arguing about NFS vs iSCSI when they really just need to accept that FC will always be superior. ;)

It's true. There was a big Compellent / Equallogic bake off at Dell back around early '13... The non-FC teams lost badly... Most took the buyout... I escaped to a Flash startup out in San Jose, but couldn't make the commute vs. real estate market work and returned to Texas in defeat. ;)

Whatever you do... If you like the storage field... Never take a cloud computing position. Most. Boring. Job. Ever! o_O:eek:
 
Top