Compression in ZVOLs with iSCSI Only

[viniciusferrao]

Hello,

I would like to know if there is any benefit in using compression in ZVOLs with iSCSI service. I can understand how compression can work in a File-based protocol, but no on a block-level protocol.

How the space will be managed from the iSCSI Client? It's aware of the compression? How it works? I'm really confused about this feature and haven't found any good explanation on this case: iSCSI with Compression.

Just to clarify things, my iSCSI have LVM inside of it.

 

[ZFS Noob]

Very good question. Thanks for asking it.

 

[jgreco]

Compression for zvols works the same way as for the rest of the pool: it probably doesn't work the way you think it does.

ZFS compresses blocks of data. Not files. Read that over a few times. Then read these words: ZFS uses a variable block size.

Now to make sense of this. Let's pretend you have a highly compressible file, two megabytes long. You're used to running "gzip" on the whole thing and having it end up as 200KB. ZFS compression DOES NOT WORK THAT WAY.

Instead, your file is broken up into ZFS blocks. FreeNAS defaults to 128KB blocks. So your file gets transformed into 16 blocks. Without compression, each one would be 128KB. However, with compression, it could be less... if a smaller blocksize can be used to store the data, then ZFS does. So for compressible data, perhaps your blocks can be compressed down to an average of 32KB. That number may not be as attractive as the per-file compression ratio. In a normal gzip-on-a-file run, the compression dictionary is potentially built over the set of the entire file contents, right? But ZFS cannot do that, because you might want to modify the blocks in the middle of the file. So the compression ratio may be less than the file-based gzip, but it is totally transparent to the user.

So this translates very directly to your question: how does a zvol work? And the answer is: "really easily." The zvol is not particularly aware of or involved in the compression; it looks like and thinks that it is just a standard volume. Under the hood, though, when ZFS is pushing blocks to and from the pool, it'll take care of the compression and decompression of the blocks transparently.

Do note that there's some interesting implications in here. You probably want a large blocksize for compression to work more effectively, but that will have performance impacts for updating single 512-byte sectors of a virtual disk created on top of the zvol because the zvol will have to read a block, decompress it to a 128K chunk, do the update, then recompress the 128K chunk and chuck it back out to the pool. iSCSI gets really fun with ZFS.

 

[viniciusferrao]

Hello jgreco, thanks for coming to the rescue again. But I still can't understand somethings. I was aware about block-level compression. But I still can't understand how this behave in relation of free space. Let's consider this example. I've an iSCSI share with any filesystem in it. How this filesystem knows this? How can I benefit from this compression (forgetting IOPS extra performance) since the upper filesystem doesn't now that a file occuped less blocks?

Thanks in advance,

 

[titan_rw]

I tested zfs gzip(max) vs a regular gzip run on a file.

A 40 gig vmdk compressed to 8.5 gig with "zfs set tank/dataset compression=gzip-9".

Using "gzip -9 < file.vmdk > file.vmdk.gz" resulting in 7.8 gig file I think.

The time difference was amazing though. Because zfs's built in gzip utilizes multiple cores, copying the file over the network took about 10 minutes (~50-60% of gigabit). The single threaded gzip -9 took multiple hours though. Didin't exactly time it, but it was horrible.

So all in all, I'm extremely happy with the 8.5 gig compressed file being how transparent and fast it is with zfs. As jgreco pointed out, iscsi zvol compression works about the same way with the caveat that random blocks are updated more often than in my case.

 

[Dusan]

I've an iSCSI share with any filesystem in it. How this filesystem knows this?

It doesn't.

How can I benefit from this compression (forgetting IOPS extra performance) since the upper filesystem doesn't now that a file occuped less blocks?

You benefit by the zvol consuming less capacity of your pool. Enabling compression may also allow you to shrink the zvol size when you delete files on the "upper" filesystem: http://nex7.blogspot.sk/2013/03/zvol-used-space.html

 

[titan_rw]

How this filesystem knows this? How can I benefit from this compression (forgetting IOPS extra performance) since the upper filesystem doesn't now that a file occuped less blocks?

It doesn't know.

For example, if you have a 100 gig zvol that you've formatted as ntfs or something, then you have a 100 gig ntfs drive. Windows doesn't know it's being compressed in the background. If the compression ratio is good, it will simply use less than 100 gigs of your zpool space.

Not sure if that helps or not.

 

[viniciusferrao]

This is what does not make sense for me :)
If I've a zpool with only one zvol for iSCSI theres, apparently, no real benefit in space optimisation.

Consider this example:
https://www.dropbox.com/s/cpcjm05srvbmkch/Screenshot 2014-01-13 20.57.29.png

So I've an 1664GB zvol serving iSCSI. Inside there is a LVM, with logical partitions with VM's inside.

If I reserve 64GB for a Virtual Disk, then I've now only 1600GB. Get more 100GB and 1500GB left. So I'll never see this "saved space" in the iSCSI. It's OK.

But theres more space in the zpool with the zvol. But it doesn't matter, since the zpool is exclusively for this iSCSI zvol... And the zvol reports it's real size to the zpool:

Code:

NAME            USED  AVAIL  REFER  MOUNTPOINT
fastpool0      1.68T   111G   148K  /mnt/fastpool0
fastpool0/vm0  1.68T   430G  1.36T  -

So in the end I don't have the free space in the zpool...

I'm missing something?

 

[cyberjock]

Nope. Your analysis is completely accurate.

You should NOT be using up more than about 90% of your pool. This is for performance reasons since ZFS is a COW file system. If you really did create a 1600GB iscsi device you will eventually hit that dreaded 90% full mark. The bad news is you'll never be able to shrink down that iscsi device later.

Ticking time bomb... tick.. tick...tick.

 

[jgreco]

You should NOT be using up more than about 90% of your pool. This is for performance reasons since ZFS is a COW file system. If you really did create a 1600GB iscsi device you will eventually hit that dreaded 90% full mark.

Oh it's much lower than that. 80-90% is the pool fill mark for regular file oriented storage. But observation is that as fragmentation increases, ZFS goes to hell at a lower percentage. I had been telling people about 60% is an approximation of the limit for a while now, but it turns out that pathological cases can cause ZFS performance to plummet even with only a little more than 10% of the pool filled.

http://blog.delphix.com/uday/2013/02/19/78/

The bad news is you'll never be able to shrink down that iscsi device later.

That kind of depends. It's doable, if you can zero out a bunch of space :)

 

[viniciusferrao]

Relax dudes! This is a testpool with 1TB disks! Don't worry...
My real pool is in other FreeNAS server with 3TB disks :)

But about my question, so what's the point of compression? If there's a better argument like: saves space for Copy-on-write. Ok... it should help. But I still can't see the point of compression (for space savings) with zvols/iSCSI.

And now I've another question: with the performance problems with iSCSI and CoW, perhaps using a file level block protocol would be a better choice to space optimisation? I'm just wondering. I've never considered using anything else for VM hosting.

 

[jgreco]

The point of compression is that it has the potential to make things faster, by reducing the amount of pool I/O, the amount of space required, etc. In a filesystem where more free space usually means greater performance, it could potentially be quite a big win.

There's nothing different about a file level block protocol. Suffers the same sorts of issues. Copy-on-write filesystems like ZFS are just natural resource hogs under certain kinds of workloads. The copy-on-write model has a lot going for it under certain use scenarios, but writing blocks in the middle of large files isn't one of them. Whether the long sequence of blocks is a file or a zvol is mostly irrelevant ... a write involves the same general steps.