Sharing ext4 to multiple initiators with iSCSI

[danj00100]

Hi,

I have a cluster that currently shares the user files via NFS, but I am currently experiencing a memory leak with NFS. This is a ubuntu problem which is reported here:

https://bugs.launchpad.net/ubuntu/+source/nfs-utils/+bug/1047566

For this reason I am investigating switching over to iSCSI to share the user data. However the current file system is ext4 and I am aware that sharing this data via a single target to multiple initiators would definitely cause data/filesystem corruption. A work around which is suggested on the wiki (http://doc.freenas.org/index.php/ISCSI) is to create multiple targets pointing to the same device so a 1:1 relationship is maintained between the targets and initiators. Here is the extract from the wiki:

NOTE: an iSCSI target creates a block device that may be accessible to multiple initiators. A clustered filesystem is required on the block device, such as VMFS used by VMWare ESX/ESXi, in order for multiple initiators to mount the block device read/write. If a traditional filesystem such as EXT, XFS, FAT, NTFS, UFS, or ZFS is placed on the block device, care must be taken that only one initiator at a time has read/write access or the result will be filesystem corruption. If you need to support multiple clients to the same data on a non-clustered filesystem, use CIFS or NFS instead of iSCSI or create multiple iSCSI targets (one per client).

My question is has anyone tested this approach and does it work? Am I right in the assumption that this approach will force the host with the targets to deal with the multi io issue. I would like some feedback on this method as I am still slightly dubious and can't see how this varies a great deal from the single target multiple initiators option?

Thanks,

Dan

 

[cyberjock]

Edit: Ok, rewrote my response. I realized I was confused. My, what a cup of coffee can do in the morning. :P

AFAIK there is no clustering support for FreeNAS at this time. But if you are having problems with NFS memory leaks, why not just try FreeNAS with NFS since its implementation is different. You wouldn't have to worry about using a clustering file system if you are using NFS since NFS will take care of the file locking/etc. A few people have asked about this, and I don't believe anyone has managed to get it to work.

I just want to say this because you are a first time poster. EXT4 isn't supported in FreeBSD/FreeNAS, nor is this a Linux/Ubuntu support forum. I think you are asking if FreeNAS will do what you want and you realize we can't fix your current problem directly. I just wanted to make sure this tidbit is clear for you in case you are confused.

I think what your best bet is if you wanted to switch to FreeNAS would be to make a FreeNAS server and move the data from your Ubuntu server to FreeNAS and serve the files via NFS on FreeNAS.

 

[jgreco]

My question is has anyone tested this approach and does it work?

Yes, it's well-tested and of course it works, but I don't think you are talking about the thing that the quoted text is talking about.

Am I right in the assumption that this approach will force the host with the targets to deal with the multi io issue.

Not "force". It just naturally does it. It isn't hard dealing with multiple initiators talking to a single target.

I would like some feedback on this method as I am still slightly dubious and can't see how this varies a great deal from the single target multiple initiators option?

Which is where it looks like you're apparently not getting the point here.

If you're using a SAN protocol (such as iSCSI), FreeNAS is acting as a SAN and just storing disk blocks. It is up to the initiator to do something interesting with that, like stick a filesystem on top of it. The target allows multiple initiators to access it, and basically doesn't know or care what you do. So if you put a cluster filesystem on top of it, like VMFS, you can have multiple reader/writer access.

If you do not have a cluster filesystem, then you must take more care. You can have an NTFS system mount an iSCSI disk, then unmount it, and then mount it on a different system. That's fine and will work. If you have two NTFS systems both mount the disk, then disaster, because NTFS isn't a cluster filesystem. But you can have two NTFS systems, each with their own iSCSI targets, because then you have two completely separate disks (and I suspect that's the part you're failing to understand). You can even do other clever things like have two NTFS systems mount the same disk, and one uses one partition while another uses a different partition. That works, basically because neither system is paying attention to the blocks that the other system is using. But you cannot have two NTFS systems both actively mounting the same filesystem.

Absent a cluster filesystem, you're probably best off with network attached storage - NAS. This pushes the task of maintaining the filesystem and controlling access to a centralized point - your NAS. As cyberjock indicated, this is probably best implemented via NFS.