NFS datastore in ESX 6.5 from Freenas performance

Mlovelace

Guru
Joined
Aug 19, 2014
Messages
1,111
I did not call those VAAI primatives. ;)
What you listed as
I believe all iSCSI devices support:
- Atomic test/set
- Full Copy
- Block Zero
- Thin Provision
- Unmap
are VAAI primitives, even if you don't call them that... that's what they are.

iSCSI has MPIO in its corner
NFS v4.1 has multipathing, though iSCSI roundrobin granularity is better in ESXi...

FC will always be superior
Well, there isn't the encapsulation with FC, but it was within a couple percentage points performance-wise.
/throws kerosene on the storage-war fire
I'm not trying to argue in favor for NFS, I don't care what protocol people use. I was only clarifying an inaccuracy about all iSCSI targets supporting VAAI.
 

rvassar

Guru
Joined
May 2, 2018
Messages
971
What you listed as
are VAAI primitives, even if you don't call them that... that's what they are.

We were discussing license-enabled VAAI extensions vs. standardized iSCSI commands. That one just happens to encapsulate the other in most cases is the crux of the misunderstanding. The one's I listed are for the most part SCSI commands (not sure on t.p...), not license-enabled extensions specific to ESX iSCSI implementation.

SCSI command reference
 

Mlovelace

Guru
Joined
Aug 19, 2014
Messages
1,111
We were discussing license-enabled VAAI extensions vs. standardized iSCSI commands. That one just happens to encapsulate the other in most cases is the crux of the misunderstanding.
The whole context of this thread is in relation to ESXi datastores. So, if the host isn't licensed for the vStorage APIs the 'standardized iSCSI commands' aren't going to be offloaded to the storage array. Which diminishes some of the argument for iSCSI. What you stated as 'all iSCSI devices supporting' is misleading in the context of ESXi, which is why I clarified, as some people might be confused.

The one's I listed are for the most part SCSI commands (not sure on t.p...), not license-enabled extensions specific to ESX iSCSI implementation.
'The one's I listed are for the most part SCSI commands'
You Listed:
- Atomic test/set
- Full Copy
- Block Zero
- Thin Provision
- Unmap

'license-enabled extensions specific to ESX iSCSI implementation'
vStorage APIs for Array Intergration:
- Atomic Test & Set
- Clone Blocks/Full Copy/XCOPY
- Zero Blocks/Write Same
- Thin Provisioning
- Block Delete (Unmap)

I'd say they are the same, which is why I brought up the licensing issue.
 

rvassar

Guru
Joined
May 2, 2018
Messages
971
'license-enabled extensions specific to ESX iSCSI implementation'
vStorage APIs for Array Intergration:
- Atomic Test & Set
- Clone Blocks/Full Copy/XCOPY
- Zero Blocks/Write Same
- Thin Provisioning
- Block Delete (Unmap)

I'd say they are the same, which is why I brought up the licensing issue.

This is getting confusing. Where did you get this list from? I have have un-licensed ESX sitting here behind me, with a iSCSI store on FreeNAS via 10GbE. Pondering how I could test this list. I could see the last two possibly being licensed features, but the first three... Those have been in SCSI since I don't know when... The early 90's? Making those licensed is extortion... ;)
 

Mlovelace

Guru
Joined
Aug 19, 2014
Messages
1,111
This is getting confusing. Where did you get this list from? I have have un-licensed ESX sitting here behind me, with a iSCSI store on FreeNAS via 10GbE. Pondering how I could test this list. I could see the last two possibly being licensed features, but the first three... Those have been in SCSI since I don't know when... The early 90's? Making those licensed is extortion... ;)
Here is the KB article https://kb.vmware.com/s/article/1021976

You can check for support by ssh'ing into the host and running esxcli storage core device vaai status get
 

Mlovelace

Guru
Joined
Aug 19, 2014
Messages
1,111
The point of the vStorage APIs for Array Integration is to offload those SCSI operations to the storage array rather than having the ESXi host complete them. The 'offload' of those SCSI operations to the array through the APIs makes them much faster to complete and frees up the host.

To quote VMware: "vStorage APIs for Array Integration is a feature introduced in ESXi/ESX 4.1 that provides hardware acceleration functionality. It enables your host to offload specific virtual machine and storage management operations to compliant storage hardware. With the storage hardware assistance, your host performs these operations faster and consumes less CPU, memory, and storage fabric bandwidth."

What does VAAI require?

VAAI requires:
  • ESXi/ESX 4.1 or later
  • Standard, Enterprise, Enterprise Plus or Remote Office Branch Office licensing for ESXi/ESX hosts
  • Storage arrays that support VAAI storage-based hardware acceleration
 
Last edited:

rvassar

Guru
Joined
May 2, 2018
Messages
971
Here is the KB article https://kb.vmware.com/s/article/1021976

You can check for support by ssh'ing into the host and running esxcli storage core device vaai status get

Interesting... Here's my FreeNAS iSCSI datastore:

Code:
naa.6589cfc00000089ff66456f17342b39e
   VAAI Plugin Name:
   ATS Status: supported
   Clone Status: supported
   Zero Status: supported
   Delete Status: supported


This is 6.0.0u3 + patches, with the "free" license. Meaning I'm registered with VMWare, but have not paid them any $$. Just ESXi, no vSphere, etc... Maybe they consider that "licensed" for VAAI purposes.

But that output seems to imply that the first three I listed are in use, or at least available to be used.
 

rvassar

Guru
Joined
May 2, 2018
Messages
971
I think we've figured out why I like iSCSI so much better than NFS... ;)

Code:
[root@<deleted>:~] esxcli system settings advanced list -o /DataMover/HardwareAccele
ratedMove
   Path: /DataMover/HardwareAcceleratedMove
   Type: integer
   Int Value: 1
   Default Int Value: 1
   Min Value: 0
   Max Value: 1
   String Value:
   Default String Value:
   Valid Characters:
   Description: Enable hardware accelerated VMFS data movement (requires compliant hardware)
[root@<deleted>:~] esxcli system settings advanced list -o /DataMover/HardwareAccele
ratedInit
   Path: /DataMover/HardwareAcceleratedInit
   Type: integer
   Int Value: 1
   Default Int Value: 1
   Min Value: 0
   Max Value: 1
   String Value:
   Default String Value:
   Valid Characters:
   Description: Enable hardware accelerated VMFS data initialization (requires compliant hardware)

[root@<deleted>:~] esxcli system settings advanced list -o /VMFS3/HardwareAccelerate
dLocking
   Path: /VMFS3/HardwareAcceleratedLocking
   Type: integer
   Int Value: 1
   Default Int Value: 1
   Min Value: 0
   Max Value: 1
   String Value:
   Default String Value:
   Valid Characters:
   Description: Enable hardware accelerated VMFS locking (requires compliant hardware). Please see http://kb.vmware.com/kb/2094604 before disabling this option.
[root@<deleted>:~]

 

Mlovelace

Guru
Joined
Aug 19, 2014
Messages
1,111
Interesting... Here's my FreeNAS iSCSI datastore:

Code:
naa.6589cfc00000089ff66456f17342b39e
   VAAI Plugin Name:
   ATS Status: supported
   Clone Status: supported
   Zero Status: supported
   Delete Status: supported


This is 6.0.0u3 + patches, with the "free" license. Meaning I'm registered with VMWare, but have not paid them any $$. Just ESXi, no vSphere, etc... Maybe they consider that "licensed" for VAAI purposes.

But that output seems to imply that the first three I listed are in use, or at least available to be used.
According to their docs they have a minimum licensing level for it at standard and above, but if it's enabled that is fantastic.
 

Mlovelace

Guru
Joined
Aug 19, 2014
Messages
1,111
Out of curiosity is the VAAI listed in the licensed features for the hypervisor only licensing version?

Features.JPG
 

rvassar

Guru
Joined
May 2, 2018
Messages
971
Out of curiosity is the VAAI listed in the licensed features for the hypervisor only licensing version?

This is ESXi only, it claims to be "vSphere 6 Hypervisor", but it's not full vSphere/vCenter. I actually don't even get a list like that. The license screen looks like this:

ESX_example.png
 

HoneyBadger

actually does care
Administrator
Moderator
iXsystems
Joined
Feb 6, 2014
Messages
5,110
@rvassar you should check to see if the commands are actually being passed down to your array, or if it's "supported, but locked away behind a vLicense" ;)

Here's a VMware KB on using esxtop to monitor your VAAI counters:
https://kb.vmware.com/s/article/2065193

Get a session going and do some stuff that you know should trip a VAAI API like make an eager-zeroed-thick disk (WRITE_SAME) copy it (XCOPY) and delete it (UNMAP)
 

rvassar

Guru
Joined
May 2, 2018
Messages
971
@rvassar you should check to see if the commands are actually being passed down to your array, or if it's "supported, but locked away behind a vLicense" ;)

Here's a VMware KB on using esxtop to monitor your VAAI counters:
https://kb.vmware.com/s/article/2065193

Get a session going and do some stuff that you know should trip a VAAI API like make an eager-zeroed-thick disk (WRITE_SAME) copy it (XCOPY) and delete it (UNMAP)

It's up 24x7, with a current uptime of 40+ days... I just checked and I have 1.1 million successful ATS events, and zero failures. Some 3000+ Zero events, with no failures. I haven't done anything lately that would invoke a clone, so those are zero'ed.
 

rvassar

Guru
Joined
May 2, 2018
Messages
971
I rebooted a FreeBSD VM, and got the ATS counter to count up ~60 more events.

Copying an entire idle VM folder in the datastore browser resulted in both ATS & Zero events, but no Clone counter activity. Similar result deleting the copy, with no Delete events counted. So it may be partially disabled. Anyone have an idea what would trigger a clone?
 
Last edited:

HoneyBadger

actually does care
Administrator
Moderator
iXsystems
Joined
Feb 6, 2014
Messages
5,110
There's a myriad of reasons why a clone/copy will fail to use the XCOPY primitive:

VAAI hardware offload cannot be used when:
  • The source and destination VMFS volumes have different block sizes
  • The source file type is RDM and the destination file type is non-RDM (regular file)
  • The source VMDK type is eagerzeroedthick and the destination VMDK type is thin
  • The source or destination VMDK is any kind of sparse or hosted format
  • Cloning a virtual machine that has snapshots because this process involves consolidating the snapshots into the virtual disks of the target virtual machine.
  • The logical address and/or transfer length in the requested operation is not aligned to the minimum alignment required by the storage device (all datastores created with the vSphere Client are aligned automatically)
  • The VMFS datastore has multiple LUNs/extents spread across different arrays
Source: https://kb.vmware.com/s/article/1021976

Or it could be that they'll give you ATS and WRITE_SAME on the freebie license, and if you want CLONE or UNMAP, you gotta pay.

Are the volumes VMFS6 with the background space reclamation job enabled?
 

rvassar

Guru
Joined
May 2, 2018
Messages
971
Are the volumes VMFS6 with the background space reclamation job enabled?

It's actually VMFS5 (v5.61). VMFS6 doesn't come in until ESX 6.5. I should probably upgrade...
 

JR Gonzalez

Explorer
Joined
Aug 29, 2014
Messages
64
I found that adding a mirrored slog helped quite a bit. I went from 200mbit to 700-800mbit by doing this. One problem I did find with NFS is that I keep getting a stale handler error in esxi 6.7. Anybody run into this?
 
Top