ZFS compression on NTFS dedupe volumes?

[mrpijey]

Hello!

I am in the process of building a new storage solution for my data. Up to now I've been using FreeNAS as a mere iSCSI initiator host to a central Windows server, and the Windows server has accessed the drives directly, handled deduplication and whatnot.

I must admit I am pretty fresh when it comes to vdevs, pools, RAIDz and all that but I've been looking at alternative storage solutions (QNAP, Synology, Windows Storage Spaces, Unraid etc) and I think FreeNAS has the best solution for me as it's cheap to setup, flexible in its configuration, lots of room for feature expansion etc.

To make things easier and more efficient I've thought about setting up FreeNAS initially with 6x 10TB drives in a pool, configured with RAIDz2 (2 drives parity, 4 drives data). Up to now I've used only individual drives, connected to my Windows server and Windows has handled encryption and deduplication on each drive, but with this solution Windows would be presented with a larger volume and it would be easier to manage, and on top of that I'll get some redundancy. I will still do backup to a separate file server that will be configured the same way. And it's easy to expand the volume as well by creating an additional RAIDz2 vdev and pool it with the first (or keep it separate presenting a second drive to Windows).

My question is however... if Windows is going to handle deduplication, should ZFS compression be enabled? Wouldn't that just waste a lot of CPU cycles trying to compress data that is deduplicated?

And are there any recommendations against my RAIDz2+iSCSI solution? The fileserver is directly connected to the Windows server by a 10Gbit connection so it doesn't need to be shared with other hosts, and since I will let Windows handle dedup I need to use iSCSI. I've considered other means like NFS etc but it just presents too much issues from the Windows side with permissions, services etc, and it works better when Windows thinks it's locally connected storage. I will be running ReFS (which also handles block level checksumming etc like ZFS does) but that overhead won't be massive anyway. And I don't think running RAID on Windows itself is a good idea since it's extremely slow if a rebuild would be needed...

Thanks!

 

[Heracles]

Hey MRP,

Your question is not clear to say the least...

about setting up FreeNAS initially with 6x 10TB drives in a pool, configured with RAIDz2 (2 drives parity, 4 drives data)

and next :

if Windows is going to handle deduplication

So who will be handling the drives and the filesystem ? FreeNAS or Windows ?

are there any recommendations against my RAIDz2+iSCSI solution?

That one is clear and so is the answer : DON'T

To server iSCSI, you need a max of IOPS. RaidZ2 does not offer much IOPS (basically as much as a single disk) so can not provide you with good performance for iSCSI. You will need to configure your pool as a Raid-10 for good performance with iSCSI.


So what are you trying to do exactly ?

You are about to deploy a FreeNAS server with its own disks and have it present an iSCSI service to your Windows host ?
Why are you using iSCSI instead of hosting the file system in FreeNAS itself and do a share at file level instead of block level ?

You mentioned dedup frequently in your post. Know that dedup is not to be used in FreeNAS because it will quickly use all your RAM for basically no benefit. Compression and proper data management / dataset configuration will do more at a much lower performance price.

Also, you did not provide us with much details about the specs of your servers. RAM, CPU, Motherboard, etc... We need that to know how well your CPU can do compression and more.

Have fun designing your system,

 

[mrpijey]

Sorry for the confusion...

The idea is to setup a RAID array on FreeNAS and present that volume to Windows through iSCSI, and then let Windows handle it as a ReFS volume. So FreeNAS will handle the actual hardware (drives) with redundancy (6 drives with raidz2), but Windows will have access to the volume exclusively through iSCSI.

Isn't the IOPS higher since it reads from multiple drives at once? It will of course be slower during writing, but that's not an issue since I add very little data to it and it's used mostly for reading. This server is also not meant to host hundreds of users, but acts as a file server for 20-30 people tops.

A lot of services and applications doesn't like accessing network shares (IIS for example, PHP is another), so it's a lot easier to have Windows handle it as a "local" drive instead. I also do deduplication on the volume, and I can't really afford a file server with several hundred GB of ECC RAM for ZFS dedup, Windows handles this fine with far less RAM. I've seen "best practises" with ZFS+dedup of between 1GB-4GB of RAM per TB storage which is a lot. I have about 120TB of data right now, and even if ZFS only handles the 40TB of raw data to dedup that would still mean at least 40GB-160GB+ of RAM initially and even more so when I expand the server with more drives.

And how will compression be better than dedup? Dedup considers the whole volume, whereas compression is per block right? Or is compression handled differently on ZFS? Currently with deduplication I can hold around 110TB of data on 32TB (4x8TB) of physical storage, and I doubt compression will do better than deduplication in this case...

The server I expect to run the new config on is a HP DL360 G8 with a hexcore Xeon E5-2430L @2GHz and initially 8-16GB of RAM, but of course I could expand both CPU and RAM if needed, but there's no way I can shove hundreds of GB of RAM into it for ZFS dedup. And I doubt mere compression will be as effective.

 

[Heracles]

Hi again,

First thing would be to make a clear difference between Raid and ZFS. There is no such thing as parity drive in ZFS. Also, ZFS does not always use all drives for a file. Should you have 6 drives and the data you write fits in 2 blocks, ZFS will store that on only 4 drives : 2 with the data and 2 with the checksums. With typical Raid, the data would be spreaded over all data drives and parity written to the last two. Also, it was Raid4 that had all of its parity on the same drive. Raid5 and 6 distribute the parity over all drives. Of course, ZFS parity is also all over the place.

Isn't the IOPS higher since it reads from multiple drives at once?

Nope. Multiple drives will not do much because all drives are required for basically every access. So at the end, your setup can work only 1 access at a time, just like a single drive does. Should you need to read something on a RaidZ2 that is 6 blocks or more (so bigger than a few kb), every drive must sync to position and retrieve their part of the data. While they are there, they can not read or write anything else.

Should your 6 drives be in an equivalent of Raid-10, a first read can be served from disk 1 while a second read on the same mirror can be served from disk 2. Also, a third read will statistically be for data on another mirror, so can be served as well at the very same moment.

iSCSI creates random read / write requests, so will be much better served from Raid-10 than RaidZ2.
20 - 30 users at once also creates random IO.
Your performance will be way better with Raid-10.

And how will compression be better than dedup? Dedup considers the whole volume, whereas compression is per block right?

Nope. Dedup is per block, just like compression.

In most systems, most of the data is unique. As such, dedup will not find that many perfect matches so will not save much. Compression will help a little with every blocks instead of saving an entire block at a time but for only a few ones.

Should your data be that redundant, better data management policy, links, dataset structure and more will also provide a better solution than dedup. In all cases, you are very aware of how important it is to avoid dedup in FreeNAS, so you got the most important point here.

 

[mrpijey]

Then it sounds it's pointless that I change how it's configured now... but I think that compression will not work as well as deduplication for my needs, I have a lot of files where 70-80% of the files are identical to eachother. I used compression before and gained massive amount of space when I switched to deduplication. I will have to do some testing. But currently I run four separate drives across iSCSI with dedup but without redundancy, and I thought it would be good to add redundancy to save some time during a downtime, and also at the same time get one volume instead of four and increase deduplication rates at the same time. But from the sound of it this would be pointless unless I massively expand everything without any real storage benefits... which I can't see me doing. RAID-10 is no option due to the massive cost increase in both harddrives and enclosures. I need to find some good option where I can have redundancy, ease of use and some speed, even if it's the same performance as I have today. I can expand on almost everything except enclosures or drives.

Thanks for the input, but I am again without a decent solution that won't toss me into the deep end of debt...

 

[Heracles]

If you really need dedup, then unfortunately you need the one feature that FreeNAS is really short on...

As of now, your performance is distributed over all of your 4 drives because they are all standalone. So as long as your requests are not all for the same drive at once, you do have the performance of multiple drives. RaidZ2 would indeed take that away.

Redundancy would surely be a good thing to add to your system, but to do it properly with FreeNAS in these conditions would require more resources than expected.

Sorry for the bad news, but better to know now then later.

Maybe this read can help you understand more about the subject...

 

[mrpijey]

Thank you for the guide, I am familiar with most of it already but I will take a look through it again :). I've also reviewed lots of benchmarks online and concluded that the speeds I would get with raidz2 would still be better than with a single drive. As I mentioned before, the load on the drives is not that very high and the benchmarks I've seen gives me more than enough for my needs (200-500MB/s on read) which is good enough. So I still think it's a viable solution as I get acceptable redundancy and still all the frills of running Windows with it. I don't have any problems with the 10G network and the iSCSI overhead is not enough to slow it down.

But of course it needs to be put to the test, verified and properly disaster evaluated before going into production. What I am going to do is setup a test server with this config (6x drives with raidz2, unfortunately not with the 8/10TB drives but with smaller drives) and test out raidz2 performance vs individual drive performance. Then I should get an approximation of the kind of performance I can expect. And I can also use a subset of data to test out Windows deduplication vs with ZFS compression/dedup and see if there are any benefits. Of course if i use ZFS compression etc then I can't use iSCSI since Windows would not be aware of any data savings so it would be unusable unless I expand the volume, so it would need to be NFS or something like it, which I am reluctant against. If i use Windows deduplication I also don't need to load up the FreeNAS servers with beefy specs as it can handle the harddrives with low power specs.

But thanks for all the tips, but I guess I will need to see the benchmarks and tests for myself. Running over 40 individual drives to Windows and managing all those volumes is not an option, and I also don't want to use the RAID features in Windows (legacy RAID or Storage Spaces as its parity is super slow even when running on bare metal), so merging all those drives into a set of volumes with parity would make my life a lot easier. And FreeNAS seems to be the only decent option, I've run it for several years now with iSCSI and never had a dropped drive or connection with it, so I don't want to change. The only alternative would be going with QNAP, Synology or the unsupported DIY Xpenology which I've also had success with on my hardware (currently running a HP Microserver with it as a backup to my real Synology NAS and both has been flawless, running RAID6 configs. )

 

[Heracles]

Hi again,

You are very welcome. As for your test, this is definitely a very good way to validate your environment. You sure can test dedup in FreeNAS, but I would not waste my time with it because dedup can be not that bad at the smaller scale test but be terrible at full scale.

As for the compression, ZFS will compress blocks after receiving them from Windows over iSCSI and decompress them before sending them back. As such, Window does not need to be aware of any compression happening at ZFS level : it sends uncompressed blocks and will receive uncompressed blocks.

As for the speed, you may be a little above one drive, but clearly not at the speed of 6 drives.

Say you have to extract a series of 24 blocks out of your RaidZ2. Each drive will need the time to sync and get its set of 4 blocks. Statistic tells us that some drives will do it faster than average, some will do it slower than average. If you know about the bell curve in statistic, let say that average is 0, min is -1 and max +1. If all drives are the same model and speed, one will measure between 0.8 and 1 ; one will be between 0.65 and .8, two will be between -0.65 and +0.65, one between -0.65 and -0.8 and last one, between -0.8 and -1. Because you need all of them to be done, you will be as slow as the slowest one, between -0.8 and -1. True you need to read only 4 blocks instead of 24, but the seek and sync time is what takes the most time.

Because on each and every request, you will drop to the slowest of the drives, the average seek and sync time will be much longer for a group of 6 than for a single drive. A single drive will do better than average about 1 per 2 request when a group of 6 will go above the average of a single drive only once per 64 requests, being below average 63 times out of 64.

Again, it is because you need all of them to complete that you will be as fast as the slowest one. The reduced amount of data to grab per disk will help, but not to the point of being as fast as that many independent drives.

If that speed is good enough for you, then go for it. Just know that there is a real risk of a bottleneck here and if you end up in a problematic zone, you will need to empty and re-do your pool over Raid-10.

Also, for your test, remember that memory caching can affect your measurement in a very significative way. Be sure to copy and read very large volume of data, well above 100% of your RAM capacity.

Good luck doing your tests,

 

[mrpijey]

Well compression of iSCSI volumes would be a waste of time really since Windows would not be able to benefit from the compression savings (as it's not aware of the underlying ZFS system), but it would nevertheless be interesting to compare the results in case I would choose to use a NFS/shared folder solution in the future. And it would be fun to compare the efficiency of Windows dedup with ZFS dedup, just for the nerdy fun ;).

Of course I don't expect the speed of 6 drives since IO activity happens on all the drives at the same time instead of just a single drive, just as with traditional RAID, but it's still above a single drive from the benchmarks I've seen. Your technical explanation is sound and also as I expected myself, but the benchmarks I've seen prove that the speed can, and often is, above the speed of a single drive. And I am not counting in caching where the speeds went far beyond a single drive speed, but long term sustained reading activity (didn't bother much with writing as it's only done from a single source and only on rare occations, my data storage being used a read-only storage).

But this is why I need to do some heavy testing, what I have available are a bunch of identical brand and model 1TB drives which I will use, setup the environment as I planned to do with the production drives and then do some real life testing, along with some heavy duty reading and writing and compare the results with a standalone setup using the same drives. A project for my just started summer vacation :).

I will surely post my results here for others to see, as I am not the only one with similar experiences and questions, and I know for a fact that iSCSI from a FreeNAS host to a Windows client is quite a common setup because FreeNAS is so good at it, and it so easy to setup :). To be continued...