ZFS Noob
Contributor
- Joined
- Nov 27, 2013
- Messages
- 129
I've delayed my testing for the last couple of weeks due to care for an ailing family member, but I've got FreeNAS on my mind today. Distractions are good sometimes.
Anyway, I'm running a Xenserver cluster of 3 hosts, and I'm evaluating a move from EqualLogic iSCSI storage to FreeNAS. Until now I've just used iSCSI because it was recommended, but now I have options and am trying to understand the implications of one choice over the other.
My test server has 72G of RAM, 4 drives in a RAID10-ish configuration, and a properly partitioned Intel 320 SSD as an SLOG. 4 GigE links in the server are set up as LAGG across a pair of switches that support LAGG, and it's working fine.
Now we get to choosing how to connect to FreeNAS. iSCSI just works, mostly. Xenserver supports it natively, and it looks fast because Xenserver does asynch writes to iSCSI. The problem is that I'm really giving up snapshotting ability with iSCSI unless I map iSCSI shares to files, and if I do that then I'll need to create a separate iSCSI share for each VM if I want to be able to rollback a snapshot for a particular VM.
If I use NFS then snapshots work a bit better. I don't know that I can recover a single VM via rollback, but at least I can clone a snapshot and move the affected VM data back with a reasonable degree to granularity. The problem I'm seeing is that NFS is slower due to asynch writes, even if I tell ZFS to treat sync=disabled for testing. I'm assuming this is due to the overhead related to synchronous writes, and the way to eliminate this would be to disable synch on the Xenserver cluster itself. Which, of course, is not recommended.
The performance penalty isn't huge, but it's noticeable. The simplest test was to reboot a VM and time it, then migrate the VM to the test array and try again. Initial results were:
Question 2: Documentation in Xenserver doesn't really cover this, but I'd assume the transaction-style writes that ZFS performs would be safer as an iSCSI store than straight iSCSI, wouldn't you? I'd also assume that async NFS writes with FreeNAS as a backend would be safer than straight iSCSI, though I think if I test enough I can get sync to be about as fast as async for running applications. Measurable differences, but unnoticeable all the same.
So where are my expectations off, and where should I be looking to test next?
Applications are database-backed web servers, mostly, with APC on the web servers and MySQL doing caching on the SQL servers. Database size is variable, but < 8 gigs of tables for the largest now.
Anyway, I'm running a Xenserver cluster of 3 hosts, and I'm evaluating a move from EqualLogic iSCSI storage to FreeNAS. Until now I've just used iSCSI because it was recommended, but now I have options and am trying to understand the implications of one choice over the other.
My test server has 72G of RAM, 4 drives in a RAID10-ish configuration, and a properly partitioned Intel 320 SSD as an SLOG. 4 GigE links in the server are set up as LAGG across a pair of switches that support LAGG, and it's working fine.
Now we get to choosing how to connect to FreeNAS. iSCSI just works, mostly. Xenserver supports it natively, and it looks fast because Xenserver does asynch writes to iSCSI. The problem is that I'm really giving up snapshotting ability with iSCSI unless I map iSCSI shares to files, and if I do that then I'll need to create a separate iSCSI share for each VM if I want to be able to rollback a snapshot for a particular VM.
If I use NFS then snapshots work a bit better. I don't know that I can recover a single VM via rollback, but at least I can clone a snapshot and move the affected VM data back with a reasonable degree to granularity. The problem I'm seeing is that NFS is slower due to asynch writes, even if I tell ZFS to treat sync=disabled for testing. I'm assuming this is due to the overhead related to synchronous writes, and the way to eliminate this would be to disable synch on the Xenserver cluster itself. Which, of course, is not recommended.
The performance penalty isn't huge, but it's noticeable. The simplest test was to reboot a VM and time it, then migrate the VM to the test array and try again. Initial results were:
- Shutdown time: 21s for EQL, 40s for NFS on FreeNAS
- Complete reboot: 51s for EQL, 1:18 for NFS on FreeNAS
Question 2: Documentation in Xenserver doesn't really cover this, but I'd assume the transaction-style writes that ZFS performs would be safer as an iSCSI store than straight iSCSI, wouldn't you? I'd also assume that async NFS writes with FreeNAS as a backend would be safer than straight iSCSI, though I think if I test enough I can get sync to be about as fast as async for running applications. Measurable differences, but unnoticeable all the same.
So where are my expectations off, and where should I be looking to test next?
Applications are database-backed web servers, mostly, with APC on the web servers and MySQL doing caching on the SQL servers. Database size is variable, but < 8 gigs of tables for the largest now.