Freenas 8.0.4 not properly SCSI-3 Persistent Reservation compliant ?

[capelink]

hi all,

I needed iSCSI and I needed reservations for cluster use - Veritas/Symantec cluster suite in particular. In case people have not used this, it includes a test utility to run against a LUN/Target from 2 nodes that tests the reservation features. This is called vxfentsthdw. I created 2 iSCSI Luns and tested on 2 nodes against one of them. The results were :

Testing blade1 /dev/rdsk/c4t11d0s2 blade2 /dev/rdsk/c4t11d0s2

Evaluate the disk before testing ........................ No Pre-existing keys
RegisterIgnoreKeys on disk /dev/rdsk/c4t11d0s2 from node blade1 ........ Passed
Verify registrations for disk /dev/rdsk/c4t11d0s2 on node blade1 ....... Passed
RegisterIgnoreKeys on disk /dev/rdsk/c4t11d0s2 from node blade2 ........ Passed
Verify registrations for disk /dev/rdsk/c4t11d0s2 on node blade2 ....... Passed
Unregister keys on disk /dev/rdsk/c4t11d0s2 from node blade1 ........... Passed
Verify registrations for disk /dev/rdsk/c4t11d0s2 on node blade2 ....... Failed

Unregistration test for disk failed on node blade2.
Unregistration from one node is causing unregistration of keys from the other node.
Disk is not SCSI-3 compliant on node blade2.
Execute the utility vxfentsthdw again and if failure persists contact
the vendor for support in enabling SCSI-3 persistent reservations

Removing test keys and temporary files, if any...


Besides the above, there were lots of errors in the freenas messages file :

Mar 23 16:01:06 freenas istgt[2113]: istgt_lu_disk.c:5228:istgt_lu_disk_queue_start: ***ERROR*** lu_disk_execute() failed
Mar 23 16:01:06 freenas istgt[2113]: istgt_lu.c:2803:luworker: ***ERROR*** LU1: lu_disk_queue_start() failed
Mar 23 16:01:27 freenas istgt[2113]: istgt_lu_disk.c:6553:istgt_lu_disk_execute: ***ERROR*** data_alloc_len(8192) too small
Mar 23 16:01:27 freenas istgt[2113]: istgt_lu_disk.c:5228:istgt_lu_disk_queue_start: ***ERROR*** lu_disk_execute() failed
Mar 23 16:01:27 freenas istgt[2113]: istgt_lu.c:2803:luworker: ***ERROR*** LU1: lu_disk_queue_start() failed
Mar 23 16:01:48 freenas istgt[2113]: istgt_lu_disk.c:6553:istgt_lu_disk_execute: ***ERROR*** data_alloc_len(8192) too small
Mar 23 16:01:48 freenas istgt[2113]: istgt_lu_disk.c:5228:istgt_lu_disk_queue_start: ***ERROR*** lu_disk_execute() failed
Mar 23 16:01:48 freenas istgt[2113]: istgt_lu.c:2803:luworker: ***ERROR*** LU1: lu_disk_queue_start() failed

The freenasserver is a quad-core i5 4Gb RAM. The targets were created as a 10g and a 20g volume off a single 1TbSATA2 disk that i had zfs'ed for testing. The initiators were on 2 Sun Blade UltraSPARC servers running solaris 10.

Any pointers/opinions would be appreciated.

Thanks

 

[FunkyZero]

I posted a similar question and haven't gotten any responses as of yet. The problem appears to be that FreeNAS and other popular packages similar to it use an implementation of iSCSI on IET instead of SCST. This is causing me issues on VMWare clustered volumes as well.
As far as my searching has come up with, there is nothing you can do.
I know there is a loadable module available for openfiler (it's not present in the normal distribution, you have to install it) that changes the system to use SCST. I am currently looking at Opensolaris as a solution since it also does ZFS and it does in fact use SCST implementation as distributed. It has tighter HCL requirements though.
If free isn't a requirement, you should also look at Starwind (Windows daemon for iSCSI) or if like myself, you wish to avoid windows at all costs, you could check NexentaStor.
I hate to have to mention other products, but there simply is no way to use FreeNAS reliably in a clustered environment since it does not have the ability to support scsi reservations. To anyone out there: If you try to use it in a VMWare HA cluster and share storage for HA, vmotion, etc... it WILL cause you never ending grief... the host machine will randomly get locked out of the LUNs and start freezing up, become non-responsive, etc. The more hosts you attach to a volume, the more prevalent the issue becomes.
I'd give just about anything to be able to say otherwise. I hope someone can chime in and tell me otherwise because I would be very excited to hear of any sort of workaround...

Here is the VMWare KB. It mentions OpenFiler, but the focus is on IET (the same that is used by FreeNAS).

http://kb.vmware.com/selfservice/mi...nguage=en_US&cmd=displayKC&externalId=1026596

I only wish I had seen this before I went through all of this work and trouble.

 

[bjohns007]

I hate to have to mention other products, but there simply is no way to use FreeNAS reliably in a clustered environment since it does not have the ability to support scsi reservations.

Here is the VMWare KB. It mentions OpenFiler, but the focus is on IET (the same that is used by FreeNAS).

http://kb.vmware.com/selfservice/mi...nguage=en_US&cmd=displayKC&externalId=1026596

I only wish I had seen this before I went through all of this work and trouble.

I totally agree w/you. I just finished implementing a FreeNAS-based solution for my home NAS/iSCSI target and just discovered the IET/SCST SCSI3 locking incompatibilities. You'd think the FreeNAS devs would place this high on the list of items to resolve since there are so many ESX boxes living inside home lab environments these days... That said, perhaps they feel that most home users don't require cluster friendly iSCSI targets? That's somewhat of a close-minded approach given that understanding clustering / HA is a requirement for almost any sysadmin job these days.

Perhaps they'll include some sort of SCST module / option when they release the next rev ... along with the latest ZFS.

 

[joeinbend]

Hi guys,
I just built out a FreeNAS server explicitly for my Hyper-V failover cluster (after finding that the commonly-used "iscsitarget" package is not SCSI-3 compliant), and FreeNAS does work. I am using the very latest version, "FreeNAS-8.0.4-RELEASE-p3-x64.iso".

Attach your iSCSI storage to each node in your Hyper-V failover cluster. Then on the node you configure your Failover Cluster Manager, go to Server Manager... Disk Management... and bring the new iSCSI disk online, and format it with NTFS. Next, in Failover Cluster Manager, you can go to Storage.. add a disk... and the iSCSI disk should be present. Now you can add it to the Cluster Shared Volumes. You will see that when you add it, the Cluster Disk will go "online".

By the way, I tried other types of Linux iSCSI targets, and although they work fine for general iSCSI storage presented up to Windows Servers, they are not SCSI-3 compliant. You know this (because of logs...), because when you try to add it to the Cluster Shared Volumes, it will add, but will always show "Failed" instead of "online".

I hope this helps! I'm happy to help if you are having trouble.

For what it's worth, my config is: two Hyper-V nodes on Server 2008 R2 SP-1, gigabit network link to FreeNAS 8.0.4-p3 server.. FreeNAS server is 8x 1TB SATA-3 disks.